Skip to content

Dev/fix twint #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
webcoderz merged 16 commits into from
Sep 5, 2022
Merged

Dev/fix twint #90

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a502953
feat(write to disk): even without neo4j
lmeyerov Sep 5, 2022
fadb48e
fix(Twint cfg): improve threading through
lmeyerov Sep 5, 2022
5397647
fix(types): missing import
lmeyerov Sep 5, 2022
74390ca
fix(output): mount output folder
lmeyerov Sep 5, 2022
5ab5cdd
infra(.gitignore): add output folder
lmeyerov Sep 5, 2022
b15ae2b
fix(twint): thinner base and use fixed twint fork
lmeyerov Sep 5, 2022
7754dba
feat(pfas job)
lmeyerov Sep 5, 2022
b3e49e8
fix(output): write to local folder
lmeyerov Sep 5, 2022
1d62d0a
feat(searches): historic and live
lmeyerov Sep 5, 2022
6d7a0f6
refactor(historic): separate historic search from usage
lmeyerov Sep 5, 2022
76782f5
feat(s3 writes)
lmeyerov Sep 5, 2022
30afc1d
feat(jobs): s3 pfas and pfsa multi
lmeyerov Sep 5, 2022
605a095
fix(pfas single job): make executable
lmeyerov Sep 5, 2022
cbd5fd0
fix(s3): handle dt inference fail
lmeyerov Sep 5, 2022
5104540
feat(retry): 3
lmeyerov Sep 5, 2022
31e584f
feat(error tolerance): try stringifying cols else drop
lmeyerov Sep 5, 2022
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(output): write to local folder
  • Loading branch information
@lmeyerov
lmeyerov committed Sep 5, 2022
commit b3e49e869d2c805c4efecf5375a0333dde319ea6
2 changes: 1 addition & 1 deletion infra/pipelines/docker/datastream-docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ services:
volumes:
- /home/lmeyerov2/neo4jcreds.json:/secrets/neo4jcreds.json:ro
- ./../../../jobs:/app:ro
- /home/lmeyerov/Work/ProjectDomino/output:/output
- ./output:/output
environment:
JOB_FILE: ${JOB_FILE:-search_by_date_job.py}
TOPIC: ${TOPIC:-covid}
Expand Down
3 changes: 3 additions & 0 deletions output/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Output folder

Job outputs in write mode go here