Skip to content

Dev/fix twint #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
webcoderz merged 16 commits into from
Sep 5, 2022
Merged

Dev/fix twint #90

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a502953
feat(write to disk): even without neo4j
lmeyerov Sep 5, 2022
fadb48e
fix(Twint cfg): improve threading through
lmeyerov Sep 5, 2022
5397647
fix(types): missing import
lmeyerov Sep 5, 2022
74390ca
fix(output): mount output folder
lmeyerov Sep 5, 2022
5ab5cdd
infra(.gitignore): add output folder
lmeyerov Sep 5, 2022
b15ae2b
fix(twint): thinner base and use fixed twint fork
lmeyerov Sep 5, 2022
7754dba
feat(pfas job)
lmeyerov Sep 5, 2022
b3e49e8
fix(output): write to local folder
lmeyerov Sep 5, 2022
1d62d0a
feat(searches): historic and live
lmeyerov Sep 5, 2022
6d7a0f6
refactor(historic): separate historic search from usage
lmeyerov Sep 5, 2022
76782f5
feat(s3 writes)
lmeyerov Sep 5, 2022
30afc1d
feat(jobs): s3 pfas and pfsa multi
lmeyerov Sep 5, 2022
605a095
fix(pfas single job): make executable
lmeyerov Sep 5, 2022
cbd5fd0
fix(s3): handle dt inference fail
lmeyerov Sep 5, 2022
5104540
feat(retry): 3
lmeyerov Sep 5, 2022
31e584f
feat(error tolerance): try stringifying cols else drop
lmeyerov Sep 5, 2022
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(Twint cfg): improve threading through
  • Loading branch information
@lmeyerov
lmeyerov committed Sep 5, 2022
commit fadb48e23282f559dabbf67a418248ab2c3e7aad
2 changes: 1 addition & 1 deletion modules/FirehoseJob.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -730,7 +730,7 @@ def search_time_range(self,
if self.save_to_neo:
logger.debug('writing to neo4j')
hydratetic = time.perf_counter()
chkd = TwintPool(is_tor=True).check_hydrate(df)
chkd = (tp or TwintPool(is_tor=True)).check_hydrate(df)
hydratetoc = time.perf_counter()
logger.info(f'finished checking for hydrate: {hydratetoc - hydratetic:0.4f} seconds')
logger.info('search step df shape: %s', df.shape)
Expand Down
4 changes: 2 additions & 2 deletions modules/TwintPool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@

class TwintPool:

def __init__(self, is_tor=False):
self.config = twint.Config()
def __init__(self, is_tor=False, twint_config=None):
self.config = twint_config or twint.Config()
self.config.Limit = 1000
self.config.Pandas = True
self.config.Hide_output = True
Expand Down