Skip to content

Dev/fix twint #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
webcoderz merged 16 commits into from
Sep 5, 2022
Merged

Dev/fix twint #90

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a502953
feat(write to disk): even without neo4j
lmeyerov Sep 5, 2022
fadb48e
fix(Twint cfg): improve threading through
lmeyerov Sep 5, 2022
5397647
fix(types): missing import
lmeyerov Sep 5, 2022
74390ca
fix(output): mount output folder
lmeyerov Sep 5, 2022
5ab5cdd
infra(.gitignore): add output folder
lmeyerov Sep 5, 2022
b15ae2b
fix(twint): thinner base and use fixed twint fork
lmeyerov Sep 5, 2022
7754dba
feat(pfas job)
lmeyerov Sep 5, 2022
b3e49e8
fix(output): write to local folder
lmeyerov Sep 5, 2022
1d62d0a
feat(searches): historic and live
lmeyerov Sep 5, 2022
6d7a0f6
refactor(historic): separate historic search from usage
lmeyerov Sep 5, 2022
76782f5
feat(s3 writes)
lmeyerov Sep 5, 2022
30afc1d
feat(jobs): s3 pfas and pfsa multi
lmeyerov Sep 5, 2022
605a095
fix(pfas single job): make executable
lmeyerov Sep 5, 2022
cbd5fd0
fix(s3): handle dt inference fail
lmeyerov Sep 5, 2022
5104540
feat(retry): 3
lmeyerov Sep 5, 2022
31e584f
feat(error tolerance): try stringifying cols else drop
lmeyerov Sep 5, 2022
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(twint): thinner base and use fixed twint fork
  • Loading branch information
@lmeyerov
lmeyerov committed Sep 5, 2022
commit b15ae2b4fbd129af2f13bbec59bf8053a3bbd5a7
9 changes: 5 additions & 4 deletions infra/pipelines/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
FROM graphistry/graphistry-blazing:v2.29.3
#FROM graphistry/graphistry-blazing:v2.29.3

FROM python:3.8

RUN export DEBIAN_FRONTEND=noninteractive \
&& apt-get update \
&& apt-get install -y --no-install-recommends supervisor \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

RUN source activate rapids \
&& pip install prefect==0.10.1 simplejson twarc neo4j boto3==1.12.39 \
RUN pip install prefect==0.10.1 simplejson twarc neo4j boto3==1.12.39 \
&& ( prefect agent install local > supervisord.conf )

COPY . .

#TODO find cleaner way to avoid talking to cloud server
RUN source activate rapids && prefect backend server
RUN prefect backend server

CMD ["./infra/pipelines/docker/entrypoint.sh"]
4 changes: 2 additions & 2 deletions infra/pipelines/docker/datastream-Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.7
FROM python:3.8
RUN apt-get update \
&& apt-get install -y --no-install-recommends git vim tor \
&& apt-get clean \
Expand All @@ -8,7 +8,7 @@ RUN pip install prefect==0.10.1 simplejson twarc neo4j boto3==1.12.39 \
pandas pyarrow urlextract git+https://github.com/homm/yurl.git@1943161973aeb3b3cf2e1e9de6671673b8356161

#RUN echo "ok6" && pip install git+https://github.com/TheDataRideAlongs/twint.git
RUN pip install git+https://github.com/twintproject/twint.git
RUN pip install git+https://github.com/graphistry/twint.git
#git+https://github.com/lmeyerov/twint.git@patch-1#egg=twint
#RUN pip install git+https://github.com/himanshudabas/twint.git@twint-fixes#egg=twint

Expand Down
2 changes: 1 addition & 1 deletion infra/pipelines/docker/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/bash

echo "Starting prefect executor daemon in foreground"
source activate rapids && supervisord --nodaemon
supervisord --nodaemon
4 changes: 2 additions & 2 deletions infra/pipelines/docker/nonrapids-Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
FROM python:3.7
FROM python:3.8
RUN apt-get update \
&& apt-get install -y --no-install-recommends supervisor \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

COPY . .
RUN pip install prefect==0.10.1 simplejson twarc neo4j boto3==1.12.39 \
pandas git+https://github.com/twintproject/twint.git@origin/master#egg=twint \
pandas git+https://github.com/graphistry/twint.git@origin/master#egg=twint \
&& ( prefect agent install local > supervisord.conf )
RUN prefect backend server
RUN ["chmod","+x","./infra/pipelines/docker/nonrapids-entrypoint.sh"]
Expand Down