Thread 105819353 - /g/ [Archived: 420 hours ago]

Anonymous
7/6/2025, 7:55:48 PM No.105819353
354484
354484
md5: b6292a7bfddb6028b852084bfbdec175🔍
Does anyone use sed and awk in 2025?
What was the idea behind all these niche unix tools with their own mini-languages?

Why modern languages, ex python, won't implement these?
Replies: >>105819514 >>105819647 >>105819694 >>105821314 >>105821363 >>105821370 >>105821857 >>105821898 >>105823255 >>105823268 >>105823302 >>105824160 >>105825580 >>105827972 >>105828030 >>105831225 >>105831325 >>105834722 >>105835840 >>105836410 >>105836855 >>105843581 >>105843786
Anonymous
7/6/2025, 8:17:50 PM No.105819514
63457456345346344
63457456345346344
md5: 707a08f5b499c7b8c5afc886c6ff6e7b🔍
>>105819353 (OP)
they are very handy and very comfy. i use them at work for various data processing tasks, when the amount of data is too large to do it by hand, or to small to involve something more advanced.
my typical usecases are retrieving smth with cli tool, storing it in temporary variable, processing it, and then feeding it as an input to something else. mainly dealing with administrative tasks on GCP. a small bash-script running inside tiny machine as a periodic cron-job can easily substitute more expensive and more resource intensive ways of doing things. i have a bunch of these small jest-werks scripts with sed and awk for various small tasks.
Replies: >>105819563
Anonymous
7/6/2025, 8:24:07 PM No.105819563
>>105819514
>they are very handy and very comfy
They are memorisation intense desu.

I work with linux for some years now, but I always need to google.
Replies: >>105819646 >>105819647 >>105822022 >>105830093 >>105831372 >>105837624
Anonymous
7/6/2025, 8:32:46 PM No.105819646
>>105819563
yeah, the syntax looks very gibberish, i honestly always try my best to write it myself, but since all the gpts became a thing - i now just feed the task and my itial code into it and ask it to expand functionality. these llms are doing very good with cli tools.
Anonymous
7/6/2025, 8:32:50 PM No.105819647
>>105819353 (OP)
They are good tools, just write in python if that's what you like.

>>105819563
Read a book, make cheat sheet etc. It's all the googling that's stopping your memory
Anonymous
7/6/2025, 8:36:52 PM No.105819694
>>105819353 (OP)
>What was the idea behind all these niche unix tools with their own mini-languages?
You needed mini-languages in order to deal with other languages, text, and mathematical operations.
I still use both but only in a limited manner. I use awk to do trivial maths for conversions and sed to edit files from the commandline I don't feel like opening in an editor.
Replies: >>105823302
Anonymous
7/6/2025, 11:36:13 PM No.105821314
>>105819353 (OP)
I use sed all the time, it's very useful for processing repetitive data
I have no idea what awk does or how it works though
Anonymous
7/6/2025, 11:41:04 PM No.105821363
>>105819353 (OP)
I use them all the time at work. I do ML so I deal a lot with CSV and text files, which I can view and manipulate using sed, grep, awk, etc. I use grep by far the most.
Anonymous
7/6/2025, 11:42:09 PM No.105821370
>>105819353 (OP)
>sed
yes
>awk
rarely, and only using basic scripts. it's really fast at processing text
Anonymous
7/7/2025, 12:21:55 AM No.105821682
I used awk until I learned about Perl in 1995. Now I don't use either of them.
Anonymous
7/7/2025, 12:45:57 AM No.105821857
1642186730449
1642186730449
md5: 381d000636c79af6efd7bbbdc8e3d4e2🔍
>>105819353 (OP)
>What was the idea behind all these niche unix tools with their own mini-languages?
As least sed was just echo cmds | ed -s initially
Anonymous
7/7/2025, 12:51:42 AM No.105821898
>>105819353 (OP)
>Why modern languages, ex python, won't implement these?
Because JSON exists.
Anonymous
7/7/2025, 1:08:10 AM No.105822022
>>105819563
>memorisation intense
if you've ever used something in the same ballpark as C, you already know 9/10ths of awk and the last 10th is conceptual understanding of how any line-by-line filter program works (and that part is optional; you can use it without input files)

It's a damn sight better than remembering all the flags of individual shell filter programs or dealing with garbage shell script syntax.
Replies: >>105823373
Anonymous
7/7/2025, 4:24:39 AM No.105823255
>>105819353 (OP)
Somewhere in the bowels of IT at my current employer, there is an expert system (think prolog) written entirely in awk.
Anonymous
7/7/2025, 4:26:52 AM No.105823268
>>105819353 (OP)
>Does anyone use sed and awk in 2025?
All the time. They're really convenient for processing text on the command line.

>Why modern languages, ex python, won't implement these?
But they do, they're just not as convenient to use.
Anonymous
7/7/2025, 4:37:38 AM No.105823302
>>105819353 (OP)
>What was the idea behind all these niche unix tools with their own mini-languages?
They are Worse is Better enshittification. The Unix shell didn't have math or string processing, so other people had to make "tools" to handle the job that normal programming languages would have built in.

>>105819694
>You needed mini-languages in order to deal with other languages, text, and mathematical operations.
In normal languages you have those features built in.
Anonymous
7/7/2025, 4:50:28 AM No.105823373
>>105822022
>It's a damn sight better than remembering all the flags of individual shell filter programs or dealing with garbage shell script syntax.
You are right about that.
It's just that to work with unix you got to memorise
>expect
>jq
>awk and sed
>krb's protocol
>socat and curl (and the used protocols)
>openssl's ever changing syntax
>sqlite
Anonymous
7/7/2025, 5:05:25 AM No.105823469
because they work so people still use them. it's not any more serious than that. and you can use them with system() and exec() and fork() and | instead of a bunch of gay python hoop jumpery
Anonymous
7/7/2025, 7:09:49 AM No.105824160
>>105819353 (OP)
Shit that only exists because of the retarded unix philosophy that everything is a text file.
>program processes the raw, real data
>formats it in a text file for humans to read
>you need to implement the formatting logic again to manually parse the text file back into the data you wanted
Why?
Replies: >>105842943
Anonymous
7/7/2025, 11:55:33 AM No.105825580
>>105819353 (OP)
>Does anyone use sed and awk in 2025?
Sure, super useful in turning tabular output into HTML reports that can ble clickable for drilling down into data.
>What was the idea behind all these niche unix tools with their own mini-languages?
Those were quick solution that fit into the Unix pipe concept, and worked well when 1 MB RAM was a lot.
Anonymous
7/7/2025, 5:47:12 PM No.105827972
>>105819353 (OP)
>Does anyone use sed and awk in 2025?
no, perl exists
the most interesting features of sed and awk is regexes but regexes exists in any scripting language, or any language for that matter, and regexes alone are insufficient for making interesting text processing tasks
Replies: >>105828017
Anonymous
7/7/2025, 5:53:48 PM No.105828017
>>105827972
>perl
The "Oh yeah, that happened" of programming languages.
Replies: >>105828092 >>105830126 >>105831018
Anonymous
7/7/2025, 5:56:02 PM No.105828030
>>105819353 (OP)
I use awk all the time for one liners, writing large scripts painful though.
I use sed syntax in vim a lot though
Anonymous
7/7/2025, 6:02:20 PM No.105828092
>>105828017
>higher-order list functions are bad
Anonymous
7/7/2025, 9:37:11 PM No.105830093
>>105819563
>They are memorisation intense desu.
We have AI for that now, just like ffmpeg commands
Anonymous
7/7/2025, 9:40:48 PM No.105830126
>>105828017
>The "Oh yeah, that happened" of programming languages.
Perl is "what if Unix people stole Lisp and pretended it was their idea."
Replies: >>105831099 >>105832660
Anonymous
7/7/2025, 11:09:33 PM No.105831018
perl 6 it'll be a success unix review
perl 6 it'll be a success unix review
md5: c1f137c975fd7b62ad2cc58d5a6f8ee5🔍
>>105828017
The best summary of Perl I have ever seen!
It'll be a success. You just wait.
Replies: >>105831056 >>105832660
Anonymous
7/7/2025, 11:14:07 PM No.105831056
1743485740798052
1743485740798052
md5: f5245be5c72320a67384793cfe8a3220🔍
>>105831018
Remember to kneel to the guy who got this kicked out as a kernel build dependency
Replies: >>105831387
Anonymous
7/7/2025, 11:17:39 PM No.105831099
>>105830126
perl is the most un-lispy language ever made what are you on about
Anonymous
7/7/2025, 11:30:02 PM No.105831225
>>105819353 (OP)
Claude grep sed awks to victory its quite nice
Anonymous
7/7/2025, 11:42:19 PM No.105831325
>>105819353 (OP)
Awk, and to a lesser extent sed, were quite influential when they came out. This was at a time when scripting languages were relatively new, your alternative to awk was writing a C program to do the same thing. In that sense awk was useful because for basic text processing it allowed you to do in a few lines what would be 20 or 30 lines of C code.

They are basically legacy tools at this point for a variety of reasons. All the UNIX tools are generally designed for processing data stored in plaintext with whitespace as a separator. These days people have learned that that format naturally leads to a variety of issues, which is why something like JSON is preferred these days. Awk and sed do have extensions for processing JSON, but even then you're better off using a modern language like Python or JavaScript since the syntax will be more readable, error handling will be better, and it will integrate better with other parts of the system.

I think at this point the main usage of awk is one liners as part of a Bash script. I still use sed occasionally for batch parsing text files.
Replies: >>105831352
Anonymous
7/7/2025, 11:47:08 PM No.105831352
>>105831325
>This was at a time when scripting languages were relatively new,
Lisp, BASIC, APL, and SNOBOL are older than C.
>your alternative to awk was writing a C program to do the same thing.
On Unix systems, but Unix wasn't popular.
Replies: >>105831371
Anonymous
7/7/2025, 11:48:46 PM No.105831371
>>105831352
>Unix wasn't popular
What? What causes people to just post clearly factually incorrect statements?
Anonymous
7/7/2025, 11:48:59 PM No.105831372
>>105819563
just read the man pages you noob
Anonymous
7/7/2025, 11:51:22 PM No.105831387
>>105831056
That guy must be a demi-god.
Total Perl eradication!
Replies: >>105838010
Anonymous
7/8/2025, 2:31:46 AM No.105832660
>>105831018
this doesn't talk about perl
>>105830126
what did perl stole from lisp exactly? closures don't count, lisp doesn't own closures
There is one thing that perl people made that was a truly new and it's its regex engine, I mean the extensions it has compared to egrep's regexes. By comparison, none of the other scripting language invented anything at all but they all re-implemented perl's regexes.
Replies: >>105835328
Anonymous
7/8/2025, 7:35:12 AM No.105834722
>>105819353 (OP)
>sed and awk
who told you to use these?
Replies: >>105834763
Anonymous
7/8/2025, 7:42:15 AM No.105834763
>>105834722
you don't go to sed/awk you end up at sed/awk
text processing tools are in a shameful state and nobody cares because they just use python for everything
Anonymous
7/8/2025, 9:15:06 AM No.105835328
>>105832660
>doesn't talk about perl
It did talk about Perl when it was published, and continued doing so for 20 years after that.
Fuck off with your revisionism and cognitive dissonance.
>truly new
>regex
Blatantly lying about regexes in a `sed` thread? This is a new low.
Replies: >>105835411 >>105835428
Anonymous
7/8/2025, 9:28:58 AM No.105835411
>>105835328
>It did talk about Perl when it was published
your image talks about perl6

>>truly new
>>regex
Did I say that perl invented regexes your fucking imbred? Not that sed was the first regex tool anyway.

>Fuck off with your revisionism and cognitive dissonance.
Fuck you, my post was specific enough if you can read.

All scripting languages use pcre essentially (not necessarily the pcre library) and it was perl that augmented the feature set of the extended regexes into what perl/pcre regexes ares. That's a fact, I am not doing any revisionism. Perl is well known for this.
Anonymous
7/8/2025, 9:31:21 AM No.105835428
>>105835328
and further more
>>truly new
>>regex
As opposed to fucking what? What did Python, Ruby, JavaScript or PHP invented/have? fucking nothing, it's all the same language features and standard library.
Replies: >>105835566
Anonymous
7/8/2025, 9:54:57 AM No.105835566
>>105835428
>What did Python, Ruby, JavaScript or PHP invented/have?
Relevance.
Replies: >>105835815
Anonymous
7/8/2025, 10:34:42 AM No.105835815
>>105835566
I accept your concession, idiot.
Replies: >>105836357
Anonymous
7/8/2025, 10:36:14 AM No.105835825
perl is a very strange thing to hold a grudge against. did larry wall rape the schizo's mom or something
Replies: >>105836357
Anonymous
7/8/2025, 10:40:16 AM No.105835840
>>105819353 (OP)
It's boomer trash, nowadays you can just ask AI to write a python script, or even a small program in Go or whatever, and it's going to be better every single time
Anonymous
7/8/2025, 12:14:14 PM No.105836357
gch
gch
md5: 90a4c7c4bcf23e05dca6cbefa26c05cf🔍
>>105835825
>>105835815
You guys sound upset. Has the line gone down or something?
Here, a relaxing read to soothe your pain: http://www.wall.org/~larry/perl.html
Replies: >>105836611
Anonymous
7/8/2025, 12:26:08 PM No.105836410
>>105819353 (OP)
sed is still commonly used for trivial search replace, just like tr.
awk is less viable in 2000 + 25 because by the time you reach for awk you could use a different programming language. Really, a bunch of stuff is basically perl adjacent where it had a decent popularity by sysadmins and workaday programmers when a server was something you could go and touch in the air conned room down the hall.
Now that servers are something you rent from Amazon and half of the shit is distributed as docker blobs anyway there's less reason to have do anything more than what daddy RedHat has configured the base install for. A programmer (tm) won't use them because it's not a webshit language and sysadmins are all retired or forcefully retired with severance.
It was better back then.
Anonymous
7/8/2025, 1:04:17 PM No.105836611
>>105836357
imagine being such a smoothbrain that you can't even develop your own opinions about programming languages. it's not rocket science, and yet...
Replies: >>105836686
Anonymous
7/8/2025, 1:15:36 PM No.105836659
I use sed -i s/too/bar occasionally or extract columns with awk. Even just a little knowledge can be useful.
Anonymous
7/8/2025, 1:19:51 PM No.105836686
>>105836611
Imagine defending an abandoned language that doesn't have a reliable way to create key-value pairs in 1995+30.
Just sad.
Replies: >>105836817
Anonymous
7/8/2025, 1:39:29 PM No.105836817
>>105836686
>doesn't have a reliable way to create key-value pairs
false and obviously so
imagine inventing bullshit arguments like that
Replies: >>105837897
Anonymous
7/8/2025, 1:44:45 PM No.105836855
>>105819353 (OP)
sed, just like tr and grep, is a standard unix tool to filter texts, therefore it'll always have its own place. Awk, on the other hand, is a whole programming language and nowadays there are many better options to choose from before reaching awk
Anonymous
7/8/2025, 3:52:54 PM No.105837624
>>105819563
I've read that o'reailly book and unix powertools or maybe couple more. I never searched how to do something with awk or sed. Also I don't get the xkcd meme about not being able to recall tar flags. It's easy af. Most tools flags make sense so you just have mnemonics over what they do

examples:
tar
-c - create
-x - e(x)tract
-v - verbose
-f - file
-z - g(z)ip
-j - bzip2 (memorize)
-J - xz (memorize)
you only need to memorize these 2 flags

find
-mindepth <num>
-maxdepth <num>
-type f : files
-type d : directories
-name <expr> : like name
-iname <expr> : like name ignore case
-not -and -a -or -o : not, and, and, or, or
-print0 : use \0 instead of \n
-exec : execute a command on match

grep and other tools share the same common flags
-E - extended regex
-e - expression
-f - file
-r - recurse
-v - non matching (reverse matches) (memorize)

awk is easy
-f file
-e expression (basically optional)
-F SEPARATOR
usage: awk '<code>' FILES...
usage: <pipeline> | awk 'code'
usage: awk -f FILE.awk : execute awk program from file

you have an implicit loop
/pattern/ {<optional body for match>}: shows only lines matching the pattern
any logical condition also applies, for example
1 : true for any line, echoes
few language variables like
NF : number (of) fields in current line
NR : number (of) row aka current line
FNR : number (of) row of many files aka current line of many files
FILENAME : name of a file
FS : field separator (how fields are split into $0 .. $NF
OFS - output field separator (how fields are printed if printed like 'print "$foo" "$bar"); space is concatenation in awk
RS : record separator i.e. line separator
ORS : output record separator
$0 : all fields aka whole line
$1 : first field of line
$3 : third field of line etc
$NF : last field of line
$(NF-2) : (last field - 2) of line etc

block BEGIN : executes before file
block END : executes after file
a program doesn't need an input such program would consist of just BEGIN { }
this is enough for most
Replies: >>105838419
Anonymous
7/8/2025, 4:28:32 PM No.105837897
Screenshot
Screenshot
md5: 502fac54004a559e75ea2b1d60ed08fc🔍
>>105836817
Cheerleading for Perl won't make it better.
You can pretend that it's a reasonable language to program in, but this only works in your tiny echo chamber. Everyone else has already moved on.
Replies: >>105840005 >>105841087
Anonymous
7/8/2025, 4:40:41 PM No.105838010
1750116118261247
1750116118261247
md5: 2ab8af0c7c3a5ab625bc9928bb8ce2e1🔍
>>105831387
I don't remember the main guy but rob landley helped rewrite the perl -es scattered across makefiles to use regular binutils
Anonymous
7/8/2025, 5:23:13 PM No.105838419
1741371886379274
1741371886379274
md5: 0a7f1b0e96a94902ab28e5ae0b7b8895🔍
>>105837624
>unix powertools
Based, can't recommend this book enough.
They need to update it to include modern tools such as fzf and jq.
Anonymous
7/8/2025, 7:56:58 PM No.105840005
>>105837897
>If I abuse the language, things go wrong.
No wonder the programming market is cratering.
Replies: >>105840067 >>105841325
Anonymous
7/8/2025, 8:03:18 PM No.105840067
>>105840005
kek, people are retards man. most people in this industry shouldn't be here (including all jeets)
Anonymous
7/8/2025, 8:14:49 PM No.105840174
A way to generate random passwords:
tr -dc a-zA-Z0-9\$\%=[]{}\(\)+- < /dev/urandom | head -c 15; echo
If you just want alphanumerics, change the tr filter to a-zA-Z0-9.
You can add it as an alias to .bash_aliases or .bashrc.
Anonymous
7/8/2025, 9:45:24 PM No.105841087
>>105837897
>Perl doesn't have a pair datatype.
I thought you were saying something was wrong with hashtables. Is that what you were talking about? There is no use case for pairs outside of emulating Lisp's conses.
It's a damn shame for any language to not have structs or unions/sum types, but "pairs"? pffffffffffft
Replies: >>105841454 >>105841808
Anonymous
7/8/2025, 10:12:05 PM No.105841325
>>105840005
>assigning the return value of a function to a variable is "abuse of the language"
Retraumatizing Perl, one sigil at a time
Anonymous
7/8/2025, 10:28:07 PM No.105841454
>>105841087
>There is no use case for pairs outside of emulating Lisp's conses.
The use case is returning a few outputs from a function without the baggage of a proper data structure like a list
Replies: >>105841502
Anonymous
7/8/2025, 10:33:57 PM No.105841502
>>105841454
that's a use case for structs and in perl all functions can return as many values as you want
Anonymous
7/8/2025, 11:09:22 PM No.105841808
>>105841087
>Is that what you were talking about?
No, hash tables work fine in Perl once created.
What is utterly broken is the way these hash tables are created, because the language does not have a reliable way of doing it.
One creates a hash by assigning a list:
%bob=('a', 1, 'b', 2) results in bob having keys 'a' and 'b'. Pretty intuitive (for the mentally retarded, at least).
But that's a useless example, most of the time you'll need *static* keys and *dynamic* values, such as %bob=('a', lower_case($name), 'b', $anon->age).
Does bob have keys 'a' and 'b' now? The answer is
>fuck knows
because 'b' will NOT be a key if lowercase($name) returns two values.
Does %bob have two keys, one key, three? More? Fuck knows, you'll have to run the code in order to find out.

To be fair, this is not that bad as far as scripting languages go, bash is even dumber and POSIX shell is downright primitive.
But bash and sh don't have delusional shills promoting them.
Replies: >>105842860 >>105844309
Anonymous
7/9/2025, 1:21:44 AM No.105842860
>>105841808
You are technically correct, but it's a completely non issue.

It's a non issue bcause this scalar/list thing concerns every single expression and statement where a LIST is involved, not just hashtable construction. You literally can't use perl for more than 5 minutes without encountering this. First you are forced to learn about sigils and even before you finish learning about them, you encouter this.

You quickly learn that although there are precise evaluation rules, you very rarely get something useful when evaluating an expression that can return a list in a context where a scalar is needed.

To be completely honnest, this is certainly the biggest flaw of perl,
(this stuff https://web.archive.org/web/20250708231543/https://www.perlmonks.org/bare/?node_id=790129)
before any syntax or type system issue, because this concerns the evaluation. But concerning your hashtable construction issue in particular, I reckon that real flaw is that the fat arrow "=>" operator should have created a scalar context for the right argument and that way you could be sure that no matter what lower_case($name) returns, you would statically know the set of keys, even if the value associated with the key "a" could be garbage.
Replies: >>105845719
Anonymous
7/9/2025, 1:34:39 AM No.105842943
>>105824160
Unix userland was originally written specifically for text processing and typesetting, that's why.
Anonymous
7/9/2025, 3:11:16 AM No.105843581
>>105819353 (OP)
Yes, millions of people every day interactively and billions automatically.
Anonymous
7/9/2025, 3:12:38 AM No.105843589
how do I git gud with sed and awk?
Replies: >>105844829
Anonymous
7/9/2025, 3:41:25 AM No.105843786
>>105819353 (OP)
>with their own mini-languages?
The language fits the abstraction and problem set extremely closely so its comfortable to use interactively.

Try scheme shell or using the Python REPL to do day to day tasks and see how unpleasant they become with a general purpose programming language.
Anonymous
7/9/2025, 5:18:17 AM No.105844309
file
file
md5: 10e77d41131372b3de0ab91bfa20d4d1🔍
>>105841808
FTFY
https://files.catbox.moe/w3fow2.gz (it should be .tar.gz)
extract it and execute make.sh

perl v5.42 but where the right operand of "=>" is evaluated in scalar context
Replies: >>105845719
Anonymous
7/9/2025, 6:31:21 AM No.105844829
>>105843589
by using them
Anonymous
7/9/2025, 9:03:33 AM No.105845719
>>105844309
Thanks. Is this change coming in v5.44?

>>105842860
>the biggest flaw of perl
I'm torn between this (context sensitivity) and the use of eval() for exception handling, but I gave up on Perl too long ago to care.
Replies: >>105846078
Anonymous
7/9/2025, 10:24:24 AM No.105846078
>>105845719
I wish but I think there is zero chance. This changes the semantics so there will be code breaking somewhere, a lot I think since it breaks *the* Moose idiom.
https://metacpan.org/pod/Moose
$ perl -E 'sub has { say join "|", @_ } has x => (is => "rw", isa => "Int");'
x|is|rw|isa|Int
$ ./perl -Ilib -E 'sub has { say join "|", @_ } has x => (is => "rw", isa => "Int");'
x|Int

>I'm torn between this (context sensitivity) and the use of eval() for exception handling, but I gave up on Perl too long ago to care.
Yeah it's not pretty. I would say that there is try/catch now but I saw that there were bugs in the first implementation (5.34 I think) so I'm not touching that even if it's fixed. The code base is the worst dumpster fire I have ever seen.