ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping #2117
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes some changes to the Shodan parser, namely:
extra.ftp.<something>.parameters, FTP parameters are collected together intoextra.ftp.featuresas a list of said features, reducing field countrsync.modulesis collectedNoValueExceptionwith a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given inhostnames, which would then be passed intosource.reverse_dnsand fail to validate as a FQDN_common_keysis moved out of the class_dict_dict_to_obj_listis introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.gvulns.CVE-2010-0001.cvss,CVE-2010-0002.cvssetc._get_firstto get the first item from a list, withNoValueExceptionraised on empty lists_get_first_hostnameto handle the first valid FQDN from a list of hostnames forhostnamesin the Shodan banner, if there is one, and givesNoValueExceptionotherwisessl.cert.serialandssl.dhparams.generator, which may return both integers and strings, are converted to stringsapply_mapping, such as reducing needless loop iterations, removing a big try-except, and adding theNoValueExceptionhandling described aboveFalse, 0) besidesNonefrom being filtered out