Timeline for What's the fastest way to generate a 1 GB text file containing random digits?
Current License: CC BY-SA 3.0
38 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 30, 2016 at 21:22 | vote | accept | posixKing | ||
| Nov 22, 2016 at 9:00 | comment | added | CodesInChaos | Even a crypto quality RNG should be able to beat 1 GB/s without a problem on a modern Intel CPU. So you typically don't need to sacrifice quality to achieve good performance. | |
| S Nov 21, 2016 at 20:14 | history | suggested | Reid | CC BY-SA 3.0 |
clarify title
|
| Nov 21, 2016 at 20:03 | review | Suggested edits | |||
| S Nov 21, 2016 at 20:14 | |||||
| Nov 20, 2016 at 2:02 | history | protected | Michael Mrozek | ||
| Nov 19, 2016 at 14:23 | answer | added | Peter Cordes | timeline score: 23 | |
| Nov 19, 2016 at 7:02 | history | tweeted | twitter.com/StackUnix/status/799870503011971073 | ||
| Nov 18, 2016 at 10:26 | history | reopened | terdon♦ | ||
| Nov 18, 2016 at 10:26 | history | closed |
mdpc Archemar Stephen Kitt terdon♦ |
Not suitable for this site | |
| Nov 18, 2016 at 10:26 | comment | added | terdon♦ | @mdpc that a question has been answered on another site is not a reason to close it here! Please don't vote to close questions because they have an answer somewhere else. Instead, post an answer with the solution that you found elsewhere. | |
| Nov 18, 2016 at 10:25 | history | edited | terdon♦ | CC BY-SA 3.0 |
You are asking for numbers between 0 and 9, not 10. ; edited tags
|
| Nov 18, 2016 at 8:41 | comment | added | Peter Cordes | That's enough to saturate main memory write bandwidth with a single core on a 3GHz CPU (dual channel DDR3 1600MHz ~= 25GB/s). | |
| Nov 18, 2016 at 8:39 | comment | added | Peter Cordes | Do you fastest to run, or fastest to type? If you mean fastest execution time, no shell command is going to be as fast as a C program, especially if you use a very cheap RNG, and take advantage of the fact that you only want one-digit numbers (so formatting integers into strings becomes trivial). You could probably vectorize a simple RNG like XORSHIFT with SSE or even AVX2 (different seeds in each vector element), and chop that up into byte elements. So I'd guess you might manage to generate results at something like 32 bytes (including spaces) every 4 clock cycles on Intel Haswell. | |
| Nov 18, 2016 at 8:35 | answer | added | sam hocevar | timeline score: 14 | |
| Nov 18, 2016 at 6:45 | answer | added | NamNT | timeline score: 1 | |
| Nov 18, 2016 at 4:29 | comment | added | njzk2 | It depends a lot on how random you want your data to be. | |
| Nov 18, 2016 at 3:56 | comment | added | HopelessN00b | How quickly can you write ~8 billion nines to a file? (Or you could use ~8 billion fours, if you prefer a mathematician's approach). | |
| Nov 17, 2016 at 21:00 | comment | added | Matthew Crumley |
This is pretty fast and RFC 1149.5 compliant: yes 4 | tr '\n' ' ' | fold -w 200 | head -c1G
|
|
| Nov 17, 2016 at 20:55 | answer | added | James Hollis | timeline score: 3 | |
| Nov 17, 2016 at 18:08 | comment | added | Toby Speight | @Mark - kernel module? For an operating system kernel? No, the fastest way is probably a GRUB module. You might need to write your own code to write to the filesystem, though... | |
| Nov 17, 2016 at 10:05 | comment | added | Nominal Animal | @posixKing: Note that although my answer is definitely tongue-in-cheek -- I'm not actually suggesting to write a C program for such a task! --, if you routinely generate such huge datasets, or you generate them often, the approach may save you time. (On my laptop, it generates 1GB of space-separated digits in about ten seconds.) However, if this is an one-off, don't even think about writing a C program for this (unless you like programming, and consider this practice or such); the shell commands and utilities achieve the task in less total time and effort spent. | |
| Nov 17, 2016 at 10:01 | answer | added | Nominal Animal | timeline score: 41 | |
| S Nov 17, 2016 at 9:39 | history | suggested | Uwe Keim | CC BY-SA 3.0 |
Formatted text
|
| Nov 17, 2016 at 9:32 | review | Suggested edits | |||
| S Nov 17, 2016 at 9:39 | |||||
| Nov 17, 2016 at 9:23 | answer | added | Michael Lorton | timeline score: 1 | |
| Nov 17, 2016 at 9:07 | comment | added | Toby Speight | You should probably say what you mean by "random" - cryptographic strength random, or is a pseudo-random sequence adequate? | |
| Nov 17, 2016 at 7:04 | history | edited | posixKing | CC BY-SA 3.0 |
added 39 characters in body
|
| Nov 17, 2016 at 5:57 | answer | added | phuclv | timeline score: 6 | |
| Nov 17, 2016 at 3:17 | review | Close votes | |||
| Nov 17, 2016 at 7:40 | |||||
| Nov 17, 2016 at 2:58 | comment | added | mdpc | What specifically is the Linux distribution and version? | |
| Nov 17, 2016 at 1:19 | comment | added | posixKing | @BillThor I already checked that, but solutions posted there add some random gibberish to the file, not numbers. I want only numbers. | |
| Nov 17, 2016 at 1:11 | answer | added | gardenhead | timeline score: 7 | |
| Nov 17, 2016 at 1:00 | answer | added | Digital Trauma | timeline score: 23 | |
| Nov 17, 2016 at 0:03 | comment | added | BillThor | See superuser.com/questions/470949/… | |
| Nov 16, 2016 at 23:16 | answer | added | Stéphane Chazelas | timeline score: 83 | |
| Nov 16, 2016 at 23:15 | history | edited | posixKing | CC BY-SA 3.0 |
added 347 characters in body
|
| Nov 16, 2016 at 23:04 | history | edited | Totor | CC BY-SA 3.0 |
edited title
|
| Nov 16, 2016 at 22:51 | history | asked | posixKing | CC BY-SA 3.0 |