About John Baker

Professional curmudgeon and relentless skeptic. "Belief" is a bullshit word. You know or you don't know!

NumPy another Iverson Ghost

During my recent SmugMug API and Python adventures I was haunted by an Iverson ghost: NumPy

An Iverson ghost is an embedding of APL like array programming features in nonAPL languages and tools.

You would be surprised at how often Iverson ghosts appear. Whenever programmers are challenged with processing large numeric arrays they rediscover bits of APL. Often they’re unaware of the rich heritage of array processing languages but in NumPy's case, they indirectly acknowledged the debt. In Numerical Python the authors wrote:

“The languages which were used to guide the development of NumPy include the infamous APL family of languages, Basis, MATLAB, FORTRAN, S and S+, and others.”

I consider “infamous” an upgrade from “a mistake carried through to perfection.”

Not only do developers frequently conjure up Iverson ghosts. They also invariably turn into little apostles of array programming that won’t shut up about how cutting down on all those goddamn loops clarifies and simplifies algorithms. How learning to think about operating on entire arrays, versus one dinky number at a time, frees the mind. Why it’s almost as if array programming is a tool of thought.

Where have I heard this before?

Ahh, I’ve got it, when I first encountered APL almost fifty years ago.

Yes, I am an old programmer, a fossil, a living relic. My brain is a putrid pool of punky programming languages. Python is just the latest in a longish line of languages. Some people collect stamps. I collect programming languages. And, just like stamp collectors have favorite stamps, I find some programming languages more attractive than others. For example, I recognize the undeniable utility of C/C++, for many tasks they are the only serious options, yet as useful and pervasive as C/C++ are they have never tickled my fancy. The notation is ugly! Yeah, I said it; suck on it C people. Similarly, the world’s most commonly used programming language JavaScript is equally ugly. Again, JavaScript is so damn useful that programmers put up with its many warts. Some have even made a few bucks writing books about its meager good parts.

I have similar inflammatory opinions about other widely used languages. The one that is making me miserable now is SQL, particularly Microsoft’s variant T-SQL. On purely aesthetic grounds I find well-formed SQL queries less appalling than your average C pointer fest. Core SQL is fairly elegant but the macro programming features that have grown up around it are depraved. I feel dirty when forced to use them which is just about every other day.

At the end of my programming day, I want to look on something that is beautiful. I don’t particularly care about how useful a chunk of code is or how much money it might make, or what silly little business problem it solves. If the damn code is ugly I don’t want to see it.

People keep rediscovering array programming, best described in Ken Iverson’s 1962 book A Programming Language, for two basic reasons:

  1. It’s an efficient way to handle an important class of problems.
  2. It’s a step away from the ugly and back towards the beautiful.

Both of these reasons manifest in NumPy‘s resounding success in the Python world.

As usual, efficiency led the way. The authors of Numerical Python note:

Why are these extensions needed? The core reason is a very prosaic one, and that is that manipulating a set of a million numbers in Python with the standard data structures such as lists, tuples or classes is much too slow and uses too much space.

Faced with a “does not compute” situation you can either try something else or fix what you have. The Python people fixed Python with NumPy. Pythonistas reluctantly embraced NumPy but quickly went apostolic! Now books like Elegant SciPy and the entire SciPy toolset that been built on NumPy take it for granted.

Is there anything in NumPy for programmers that have been drinking the array processing Kool-Aid for decades? The answer is yes! J programmers, in particular, are in for a treat with the new Python3 addon that’s been released with the latest J 8.07 beta. This addon directly supports NumPy arrays making it easy to swap data in and out of the J/Python environments. It’s one of those best of both worlds things.

The following NumPy examples are from the SciPy.org NumPy quick start tutorial. For each NumPy statement, I have provided a J equivalent. J is a descendant of APL. It was largely designed by the same man: Ken Iverson. A scumbag lawyer or greedy patent troll might consider suing NumPy‘s creators after looking at these examples. APL’s influence is obvious. Fortunately, Ken Iverson was more interested in promoting good ideas that profiting from them. I suspect he would be flattered that APL has mutated and colonized strange new worlds and I think even zealous Pythonistas will agree that Python is a delightfully strange world.

Some Numpy and J examples

Selected Examples from https://docs.scipy.org/doc/numpy-dev/user/quickstart.html Output has been suppressed here. For a more detailed look at these examples browse the Jupyter notebook:  NumPy and J Make Sweet Array Love.

Creating simple arrays

 # numpy
 a = np.arange(15).reshape(3, 5)
 NB. J
 a =. 3 5 $ i. 15

 # numpy
 a = np.array([2,3,4])
 NB. J
 a =. 2 3 4
 # numpy
 b = np.array([(1.5,2,3), (4,5,6)])
 NB. J
 b =. 1.5 2 3 ,: 4 5 6

 # numpy
 c = np.array( [ [1,2], [3,4] ], dtype=complex )
 NB. J
 j. 1 2 ,: 3 4
 # numpy
 np.zeros( (3,4) )
 NB. J
 3 4 $ 0
 # numpy - allocates array with whatever is in memory
 np.empty( (2,3) )
 NB. J - uses fill - safer but slower than numpy's trust memory method
 2 3 $ 0.0001 

Basic operations

 # numpy
 a = np.array( [20,30,40,50] )
 b = np.arange( 4 )
 c = a - b
 NB. J
 a =. 20 30 40 50
 b =. i. 4
 c =. a - b
 # numpy - uses previously defined (b)
 b ** 2
 NB. J
 b ^ 2

 # numpy - uses previously defined (a)
 10 * np.sin(a)
 NB. J
 10 * 1 o. a
 # numpy - booleans are True and False
 a < 35
 NB. J - booleans are 1 and 0
 a < 35

Array processing

 # numpy
 a = np.array( [[1,1], [0,1]] )
 b = np.array( [[2,0], [3,4]] )
 # elementwise product
 a * b

 NB. J
 a =. 1 1 ,: 0 1
 b =. 2 0 ,: 3 4
 a * b

 # numpy - matrix product
 np.dot(a, b)

 NB. J - matrix product
 a +/ . * b  
 # numpy - uniform pseudo random
 a = np.random.random( (2,3) )
 NB. J - uniform pseudo random
 a =. ? 2 3 $ 0
 # numpy - sum all array elements - implicit ravel
 NB. J - sum all array elements - explicit ravel
 +/ , a
 # numpy
 b = np.arange(12).reshape(3,4)
 # sum of each column
 # min of each row
 # cumulative sum along each row
 # transpose

 NB. J 
 b =. 3 4 $ i. 12
 NB. sum of each column
 +/ b
 NB. min of each row
 <./"1 b
 NB. cumulative sum along each row
 +/\"0 1 b
 NB. transpose
 |: b

Indexing and slicing

 # numpy 
 a = np.arange(10) ** 3 
 a[ : :-1]   # reversal

 NB. J
 a =. (i. 10) ^ 3
 2 { a
 (2 + i. 3) { a
 |. a

Government Shutdown and Rapture 2018

It’s 2018. 2018 has prime factors of 2 and 1009, e.g. 2018 = 2 * 1009.

Did anything happen in the years 2 and 1009?

  1. In the year 2 Venus and Jupiter were in conjunction. Some speculate this may have been the “Star of Bethlehem.”
  2. In 1009 the Church of Holy Sepulcher was destroyed.

How odd, the prime factors of 2018 yield two Jesus related events, separated by over a thousand years that are mysteriously tied to his birthplace. Evidently, the Star of Bethlehem marked the beginning, 1009 marked the middle, and 2018 is the end! Repent, the rapture is upon us! Better buckle up non-gender specific bovine control officers (cowboys) 2018 is going to be rough.

Just how rough you ask? Imagine hordes of underachieving and mostly useless bureaucrats forced to take an impromptu paid holiday1 because even more useless legislators cannot agree on how to spend other people’s money. Let’s hope the government stays shutdown down until the imminent rapture renders it redundant.

My calculations cannot pinpoint the exact time and date of the rapture. My guess would be the first2 blood moon of the year, January 31, 2018. Get your affairs in order and divest yourself of your worldly bitcoins.3 Use the address encoded in the QR graphic of this blog.

Hey: If you think this micro-epiphany is batshit crazy take a look at this! It’s getting to the point where it’s no longer possible to be satirical because somewhere on the intertubes you’ll find true believers that go way beyond satire.

  1. None of the furiously furloughed will miss a single damn paycheck. Where can I get a job that pays me to sit on my ass and whine about whatever the idiot talking point of the day is?
  2. 2018 has two total lunar – blood moon – eclipses. If we’re not raptured on January 31st wait until July 27th
  3. Sky Fairy consultants, also known as prophets, have high overhead and need substantial donations to maintain their close connections with the divine.

Dear Apple Pay Stop Harassing Me!

Dear Apple Pay stop harassing me! I will never sign up for you and no amount of iOS app-nagging will change that. You’re pathetic interruptions merely remind me why I “resist” installing every freaking iOS upgrade the Apple mothership foists on its hapless phone users.

I use an iPhone1 but I am not a member of the bleating iSheep. I’ve slowly, one unwanted upgrade after another, developed an intense loathing of Apple’s holier than thou, we know what’s best for the iIdiots, attitude. You’re beginning to remind me of the Hildabeast. Remember that foul creature and her well-deserved fate!

Loathing aside, I have a perfectly logical reason for refusing Apple Pay. Unlike the bleating iSheep, I see what you are trying to do. You want to become a financial “middleman.” You want to skim an ever-increasing percentage of every transaction your well-trained herd of iSheep makes. I admire your larcenous spirit but if you haven’t noticed, we have enough parasitic financial rent-seeking middlemen on this insane planet. Part of the allure and promise of cryptocurrencies is that they show a way to rid the universe of skimming scum like central banks, government fiat, rapacious money transmitters, exchange controls and abominations like Apple Pay.

Don’t ever darken my day with another “activate Apple Pay now” message again. You have been warned!

  1. Unless there are big changes at Apple this is my last iPhone. I want a device that I absolutely control. “If you don’t control it you cannot trust it.” Any phone that I cannot even be sure is off does not qualify. Burners may be my only option.

Review: The Way We Die Now

The Way We Die Now is not the best book I’ve read this year but it may be the most important. In Seamus O’Mahony’s opinion, modern society has forgotten how to deal with death. There are many reasons for this, the collapse of religious belief, the demolition of the extended family, the triumph of the scientific and rational worldview, even our delusions of curing death “real soon now” contribute to our collective denial. Yet death persists. Death remains absolute, sovereign, implacable, terrifying, “majestic and cruel.” Even if we realize our singularity fantasies and greatly extend life death will never be banished. Even the gods die! We must face death, but must we turn it into a carnival of “medical excess?”

I have seen medical excess. My mother was diagnosed with Stage IV Glioblastoma: a form of brain cancer that is so deadly it’s been nicknamed the terminator. Actually, the terminator is flattered by the comparison. Some survived their encounter with fictional terminators. Nobody survives stage IV Glioblastoma: “there is no stage V.” When I heard mom’s diagnosis I looked for actuarial survival statistics. Credible statistics for common fatal diseases are harder to track down than you might expect. I eventually found a paper that cast survival times in a useful form. Median survival was less than three months for younger and healthier patients than my mother. She died about two months after her diagnosis – right on statistical schedule. The universe does not make personal exemptions.

Her death was inevitable, but the expensive, futile, painful and isolating medical gauntlet she endured was not. She just wanted to go home, perhaps to “turn her head to the wall,” perhaps to binge on The Big Bang Theory – she still enjoyed a few silly shows. It doesn’t matter what the dying choose to do with their remaining hours, but it sure as hell matters that we honor their choices and the Way We Die Now makes a compelling case that we are failing “to be brave.” I know I acquiesced to the medial default for my mother; I still feel I should have fought harder for what she wanted.

According to O’Mahony, the medical default is full intervention even when it’s pointless and wasteful. He also notes that doctors are in a no-win situation. If they suggest doing nothing they’re accused of euthanizing patients. If they go full interventionist Rambo they’re inflicting needless suffering and profiting from the dying. Both extremes often end up in court, as if we could fix death with more litigation. Obviously, something in the middle is the best course and O’Mahony argues that doctors should not set the middle course.

Our infantile society needs to grow up and face death like adults. Nothing makes our magical thinking about death clearer than Somerset Maugham’s1 observations about a “dog’s death.” Maugham hoped he was lucky enough to die a dog’s death! A dog’s death is meant to be a horrible thing but is it really worse than human medical excess? When it comes to sick animals we are clear-headed and compassionate. We don’t subject them to futile treatments, we make them comfortable and take away their pain. I once had a cat that came down with pancreatitis. She wasted away on the top of our fridge until one day we took her to the vet. Her death was calm and without terror. My cat had a better death than my mother. I suspect many pets die with greater dignity than their owners. This is fundamentally wrong and we all know it.

There are no easy answers; it sucks to be mortal. We can’t say until we face it ourselves how we should die so how can we dictate to others? I only hope that when my time comes I have it within me to follow the one bit of advice O’Mahony offers that may apply to all us – “be brave.’

  1. The Way We Die Now relates many stories about “celebrity” deaths.

The Mass Kill as Performance Art

It’s been almost a week since the Las Vegas Mandalay Bay massacre and the idiot media is still looking for “a motive.” They remind me of O.J.’s fruitless search for the “real killer.” I don’t watch the alphabets, except when trapped in airports or, with increasing annoyance, in my employer’s cafeteria so I’ve missed days of mindless speculation but my limited TV sampling confirms what’s easily gleaned from more efficient news sources. The killers “motive” is still unknown and the authorities are still looking.

In all the blather about the mysterious highly organized and thoughtful killer, it’s never occurred to anyone (on TV anyway), that the lack of an apparent motive is exactly what the killer hoped to achieve.

Consider the usual mass media slaughter script. An individual, or group of individuals, attack and murder a sufficiently1 large number of “innocents.”

If, as is often the case, the attackers are well-known terrorists they will typically gloat and issue more threats. Public threats trigger the idiot media’s “analysis.”

If the terrorists are Jihadis the idiot media will downplay the attack while issuing stern warnings about not jumping to racist conclusions about an “entire religion.” Then, when forced by competing news organizations like Fox or right-leaning bloggers, they will join the fray and condemn the killers while searching for a way to blame Trump.

If the killer is black and the victims are black — well this isn’t news! Press the ignore button and complain about anything that can be plausibly blamed on Trump. And, fortunately for the idiot media, that well will never run dry.

If the killer is white and the victims are mostly white (bingo for Las Vegas), spend a day or two glorifying the murderer. Review his2 typically pathetic and meaningless life while running candlelight vigils, peace garden plantings, and out-of-tune Kumbaya-a-thons in the background. In more sober moments touch on the known motive” for the mayhem. Tut-tut the violence, reassure moronic viewers that violence is never the answer. Play a few bars of Imagine then, when appropriate mourning turns to anger, use the “known motive” to pivot to what the idiot media wanted to talk about from the beginning: gun confiscation.3

The sooner the idiot media can get “the motive” out of the way the sooner they can get down to one of their favorite topics. Sadly, the Vegas killer, being an intelligent and detail oriented psychopath, anticipated this and left no clear motive forcing the idiot media to fixate, like a dog licking its ass, on “the motive” for day, after day, after day. Every day spent discussing “the motive” inflates the killer’s infamy. He’s already the most famous mass killer in recent history.

Real villainy requires incomprehensible dread and clear motives eliminate dread. When Jihadis kill it’s no big deal. Their sky fairy manual exhorts true believers to behead, enslave and tax infidels. It’s hard to imagine a violence-free way to realize such goals. When black hating psychopaths open fire in churches the idiot media have ready-made dread dispelling explanations. If you can quickly explain an atrocity it almost ceases being atrocious. But, if the horror can never be explained, if it sticks in your existential craw, it remains a source of terror forever. This is what the killer was really aiming for.

I view the Las Vegas massacre as deadly performance art. The killer has more in common with “artists” that drop their pants in public and pee on crucifixes than cause-driven revolutionaries or garden variety psychopaths. What exactly motivates public “performance” crucifix urination? To even ask the question is to mock it. For one deadly night the Vegas killer staged a performance that upstaged all the other Vegas shows and unlike another showing of Cirque du Soleil or Menopause the Musical his performance will be remembered forever. The mass kill is a new art form and the idiot media is its biggest patron.

  1. Sufficiently large is an ever-increasing natural number that is now greater than ten.
  2. The overwhelming majority of mass killers are male.
  3. It’s impossible to have an intelligent debate about guns in the US is because neither side is willing to discuss their ultimate goals or take responsibility for their positions. Gun controllers secretly want the second amendment appealed and all existing guns confiscated. Nothing else will satisfy them. They naively think an unarmed society will never be abused by the state. Gun holders often point out that the second amendment was never about creating a safe and secure society but about erecting deadly barriers to government tyranny. Unfortunately, they’re unwilling to admit that a heavily armed populace will result in large numbers of firearm deaths. US gun deaths are an order of magnitude greater than comparable western societies: that tree of liberty is more bloodthirsty than expected.