Learning the Crystal langauge and Lucky web framework

Crystal is a rising programming language with the slogan “Fast as C, Slick as Ruby”.  It has some compelling features that make it more attractive than other modern language attempts like Go. You really can program in a Ruby-like language and achieve software that performs with the speed of a compiled language.

But the greatest advantage of Crystal, that I have experienced so far, is that it provides type-safety without excessive declarations as you would see in Java. It does this through program-wide type inference. So, if you write a function like this:

def add(a, b)
  a + b
end

add(1, 2) # => 3, and the returned type is Int32
add(1.0, 2) # => 3.0, and the returned type is Float64

You get type-safe duck-typing at compile-time. If a method isn’t available in a type, you’ll find out at compile-time. Similarly, the type of a variable can be inferred from what you assign to it, and does not have to be declared.

Now, let’s say you never want to see nil as a variable value. If you declare the type of a variable, the compiler will complain at compile-time if anything tries to assign another type to it. So, this catches all of those problems you might have in Ruby or Javascript with nil popping up unexpectedly as a value and your code breaking in production because nil doesn’t have the methods you expect.

There are union types. So, if you want to see nil, you can declare your variable this way:

a : String | Nil

a  : String? # Shorthand for the above.

Crystal handles metaprogramming in several ways. Type inference and duck typing gives functions and class methods parameterized types for free, without any declaration overhead. Then there are generics which allow you to declare a class with parameterized types. And there is an extremely powerful macro system. The macro system gives access to AST nodes in the compiler, type inference, and a very rich set of operators. You can call shell commands at compile-time and incorporate their output into macros. Most of the methods of String are duplicated for macros, so you can do arbitrary textual transformations.

There is an excellent interface to cross-language calls, so you can incorporate C code, etc. There are pointers and structs, so systems programming (like device drivers) is possible. Pointers and cross-language calls are “unsafe” (can cause segmentation faults, buffer overflows, etc.) but most programmers would never go there.

What have I missed so far? Run-time debugging is at a very primitive state. The developers complain that LLVM and LLDB have changed their debugging data format several times recently. There’s no const and no frozen objects. The developers correctly point out that const is propagated through all of your code and doesn’t often result in code optimization. I actually like it from an error-catching perspective, and to store some constant data in a way that’s easily shareable across multiple threads. But Crystal already stores strings and some other data this way. And these are small issues compared to the benefits of the language.

Lucky

Paul Smith of Thoughtbot (a company well-known for their Ruby on Rails expertise) is creating the Lucky web framework, written in Crystal and inspired by Rails, which has pervasive type-safety – and without the declaration overhead as in Java.

The point of all of this is that you can create a web application as you might using Ruby on Rails, but you won’t have to spend as much time writing tests, because some of the most common problems of Ruby code are taken care of by the type system. And the combination of exceptions and type-safety does an excellent job of getting rid of most of the function return error checking I’d have to write in other languages. When you want to check for a nil rather than catch an exception, there are method versions suffixed with a ? which provide that.

Learning Crystal and Lucky, since I’m already a Rubyist, wasn’t difficult, but I took about two days including finding some bugs in Lucky and learning some non-obvious things about the language. Like it’s better not to declare the types of things a lot of the time. Rather than look up that the type of something was Lucky::AdmittedField, I could just declare the name of an argument that used it and go on with my life, and the compiler would take care of things.

The biggest problem with Lucky right now, in its pre-1.0 state, is that there is no API documentation. There are tutorial guides that tell you how to do most things, but I found myself exploring the Lucky code several times.

I am porting an application I’d written in Ruby for a new startup to Crystal and Lucky, to see if I can have more comfortable development with fewer run-time errors. If this works, I’ll have a large production application to better evaluate the language and framework.

Somewhere in the world there is someone in love with Node who is asking why I don’t use that. Javascript isn’t a particularly elegant language. Attempts to pretty it up like Coffeescript fall short of what you really should see in a modern language.

The advantage of Node is that the native IO framework is non-blocking. Some Node enthusiasts don’t realize that almost every other web framework and server does non-blocking IO to handle the web requests, and you don’t have to concern yourselves with that. But you still have blocking by default for database queries, file I/O, and your calls to other services in the cloud. Crystal library authors could provide non-blocking I/O with promises for this, but the developers haven’t seen a good reason to do so. Crystal uses Fibers for concurrency (and will get multithreading). Fibers start with a 4K stack, and are so inexpensive that a 64-bit processor can realistically provide thousands of them per process. Having a straight-line logical flow through I/O rather than many event-handling blocks (probably nested) means more readable and maintainable code. The overhead of fibers seems a low cost for that.

And finally, one thing I won’t ever miss is a JIT compiler as in Java and Javascript, and its complexity. The architecture portability reasons elucidated when Java was created were never nearly so big an issue as expected – even on Android phones. It works to have it in browsers, but even there the future focus is on Webassembly, a bytecode that runs inside of the Javascript engine, which will be compiled from various other languages.

Open Source vs. Munitions Export Restrictions – Announcing Open Research Institute, Inc.

In the United States, some technologies are considered “munitions”. Space satellites, rocketry, digital voice encoding, and encryption are among them. Depending on the technology, they are regulated by the Department of State under ITAR, or the Department of Commerce under EAR.

The Open Source cryptography community worked out how to operate in compliance with ITAR and EAR a decade or more ago, filing several lawsuits against the U.S. Government on the way. Unfortunately, the Amateur Satellite and Space community hasn’t been as up-to-speed. It’s time for that to change.

In January, Michelle Thompson W5NYV and I decided to form a new non-profit corporation specifically structured for the purpose of operating international Open Source development of projects that might otherwise be considered to be munitions under ITAR and EAR. To do this, we make use of carve-outs in ITAR and EAR for published information.

Ben Hilburn (of GNU Radio fame) joined Michelle and I on the board of directors.

Michelle is leading the Phase 4 Ground Station project, an Open Source project to create a digital space communications system. Phase 4 Ground and my Open Cars research will be the first projects under our new organization, Open Research Institute, Inc.

We are now incorporated in California and are filing our registration with the State as a charitable organization today. We have contracted a firm to prepare our 501(c)3 tax-exempt non-profit application with the Internal Revenue Service.

Read about Open Research Institute.

Bruce Perens Seeks Mandatory Award of Legal Fees For His Defense in Open Source Security, Inc. and Bradley Spengler v. Bruce Perens

In August, Open Source Security, Inc. brought a lawsuit against me in federal court, seeking over $3 million, and later added Bradley Spengler as a plaintiff. Open Source Security and Mr. Spengler sued me because they disagreed with my blog posts and Slashdot comments which expressed my opinions that their policies regarding distribution of their Grsecurity product could violate the GPL and lead to liability for breach of contract and copyright infringement.

This lawsuit should never have been brought.  My blog post and Slashdot comments are first amendment speech protected by the California anti-SLAPP law.  While Open Source Security and Mr. Spengler were free to disagree with my opinions, they were not free to sue me to try to stop me from expressing them.  The anti-SLAPP law requires recovery of attorneys’ fees and costs–precisely to deter actions like this one that chill the exercise of first amendment rights.  Fee recovery also encourages private representation in anti-SLAPP cases so that defendants may obtain quality representation without being bankrupted.  Thus, the law provides for successful defendants to recover fees and costs from plaintiffs who bring meritless suits.

The court has now ruled in my favor that Open Source Security and Mr. Spengler’s claims are not meritorious.  Today, I am asking the court to award my legal fees and costs so far, which will go to O’Melveny, the law firm I retained.

The cost of my defense well exceeded half a Million dollars.

When I got sued, I called upon the best attorney I know in the Open Source world, Heather Meeker, who I have known for 20 years and worked with professionally. Heather is the author of Open (Source) for Business: A Practical Guide to Open Source Software Licensing — Second Edition and three other books.  Heather formed a team including Melody Drummond Hansen, a Silicon Valley partner with experience in Open Source and defamation matters, Cara Gagliano, an associate focused on copyright and speech issues, and two bright first-year associates, Marissa Rhoades and Eric Ormsby .

I sought this high-quality team because the stakes were high.  Open Source Security and Mr. Spengler sought at least $3 Million and also attacked my professional reputation and integrity.  While the fees incurred in this case are significant, they are based on the standard rates charged for attorneys of similar experience and caliber.  In my experience, this team is well worth it.  Also, the amount of fees reflect Open Source Security’s and Mr. Spengler’s litigation tactics, which unnecessarily increased the time spent defending this case by forcing responses to the many filings and shifting theories they pursued.

Had Open Source Security and Mr. Spengler not filed their suit, they would not be facing this expense at all.  For example, had they engaged in the spirit of reasoned debate rather than filing a lawsuit, the fees could have been avoided entirely.  And had they not greatly complicated the suit, the fees would be much smaller.  Plaintiffs now also are appealing the case, which will further increase my legal costs.

Fee awards under anti-SLAPP are mandatory and they are meant to deter unnecessary suits like this one–to protect everyone’s freedoms to engage in public debate without fear of being sued.

Generally, I would prefer not to talk about a lawsuit in progress. My desire to keep the Open Source community informed is my reason for making this statement.  Unfortunately, I will not be able to make any further statements about the case for now.

ARRL Suspends Controversial Director Confidentiality Requirements

At its January 18 meeting, ARRL suspended some of the controversial director confidentiality requirements. These requirements first made news when ARRL publicly censured director Richard Norton N6AA, apparently for publicly discussing at an ARRL conference the requirements themselves and the board’s handling of the matter.

The board resolved that:
The entire Code of Conduct must be reviewed by the ARRL Officers, Directors and Vice Directors with a deadline for completion of a final draft version 60 days in advance of the July 2018 meeting of the ARRL Board of Directors and be reported at that meeting.

 

So, we will need to move in the upcoming month to inform ARRL directors of our sentiments regarding the code of conduct, while this draft is being created.

 

The board moved to delete one section of the existing code of conduct and suspend another:

1. Confidentiality 6. C. delete the following sentence: “A Board member may not, in disclosing anything about the Board’s deliberations, discuss or disclose the votes of the Board or of individual Board members (including his/her own) unless the Board has previously made the votes public”.
2. Suspend all of Section 8. “Support of Board Decisions.”

 

The action by the board was much better than the previous rather unresponsive A Note to Members from ARRL President Rick Roderick, K5UR. Roderick didn’t venture to apologize in any way for the new code of conduct and its application to Director Norton, which at least some of the members found disturbing. One would hope that Roderick, re-elected as president at the same meeting, could man up and say “I’m sorry”, since the board did indeed look into the issue and take action just days after his unresponsive editorial, and must have already had the item in its agenda as Roderick wrote his piece.

 

These are the sections that are deleted or suspended:
6(c)A Board member may not, in disclosing anything about the Board’s deliberations, discuss or disclose the votes of the Board or of individual Board members (including his/ her own) unless the Board has previously made the votes public. Nor shall any Board member falsely characterize the positions, policies or decisions of the Board or the points of view taken by any member of the Board with respect to them.
8. SUPPORT OF BOARD DECISIONS: A Board member must accept and publicly support Board decisions.
a. A Board member, as a leader in Amateur Radio, is encouraged to be an ambassador and an advocate for ARRL and, subject to the Confidentiality Standard of this Code of Conduct, to publicly promote the activities and actions of the organization with the ARRL membership. In doing so, a Board member must act at all times faithfully to the intent of the Board as expressed in its official statements, and should not reinterpret or re-characterize the Board’s actions to reflect his/her own view or the views of any other Board Member.
b. While having the right and responsibility to exercise independent judgment and to express dissenting opinions during Board deliberations, a Board member also has the obligation outside the Boardroom to respect and support final decisions of the Board, even when the Board member dissented from the majority view.
c. A Board member who does not support a Board decision may express his/her opposition within the Board in an appropriate manner.
d. A Board member must not take actions publicly or with respect to the ARRL membership that have the purpose or effect of undermining or discrediting the decisions or actions of the Board.
e. If a Board member is ultimately unable to accept a Board decision and is unable to influence a change, the Board member should consider voluntarily resigning his/her position on the Board.
f. A Board member may not publicly oppose a Board action prior to the effective date of his or her resignation from the Board.

SpaceX Expendable Rocket Fails

SpaceX expendable rocket fails to be expended

Today’s Sescom launch by SpaceX failed spectacularly when the expendable first stage failed to be expended, surviving after soft-landing in the ocean. An embarrassed SpaceX had no choice but to attempt to tow the rocket to shore, lest it reveal ITAR-restricted secrets and other proprietary information of SpaceX to anyone who cared to salvage it.

SpaceX Zuma Launch Might Not Have Been a Real Satellite

I wrote a while back that the Zuma satellite mission might not have been real. Well, SpaceX launched it yesterday, Sunday January 8, and there is still no indication that a real satellite was launched. While the Zuma mission is secret, rumors today are that the mission failed.  See 1 2 3 4 and the best article so far. SpaceX claims that nothing went wrong on their side. My surmise is that this mission was for the purpose of evaluating SpaceX readiness to perform launches for government intelligence, and like the Falcon Heavy demo (which is carrying Elon’s Tesla Roadster) might not have been carrying a functional payload.

Understanding the Red Hat – IBM – Google – Facebook GPL Enforcement Announcement

On November 27, Red Hat, IBM, Google, and Facebook announced that they would give infringers of their GPL software up to a 30-day hold-off period during which an accused infringer could cure a GPL violation after one was brought to their attention by the copyright holder, and a 60 day “statute of limitations” on an already-cured infringement when the copyright holder has never notified the infringer of the violation.  In both cases, there would be no penalty: no damages, no fees, probably no lawsuit; for the infringer who promptly cures their infringement. I’ll discuss this in a question-and-answer style:

Q: Does this change any court case involving the GPL that is presently in progress? A: No. Red Hat, IBM, Google, and Facebook are not known to be involved in any recent cases of enforcing their copyrights on any software under the GPL license. Red Hat once suggested that it could enforce the GPL against patent aggressors, but I am not aware of any cases in which they have done so.

Q: Does this change the way that GPL license on Linux or any Open Source program will be enforced? A: That’s unlikely. It is not in the interest of those four companies to become involved in enforcement of the GPL license in the future. Thus, their promise to allow an infringer to cure their infringement within certain time limits is irrelevant. Red Hat and IBM would rather sell Linux systems than make their customers afraid by bringing lawsuits. Google and Facebook have bigger fish to fry.

Q: Does anyone else have to provide the same time limits? A: No. The companies involved in the announcement are not the entities that bring GPL enforcement lawsuits. There are thousands of copyright holders in the Linux kernel who could theoretically bring a lawsuit, and perhaps one hundred thousand authors of GPL-licensed software other than Linux (but which might be included as part of a typical Linux system). None of those parties are compelled by today’s announcement. But in general those parties have been voluntarily waiting much longer for infringements to be cured without penalties.

Q: Does their announcement comply with the Principles of Community-Oriented GPL Enforcement drafted by the Software Freedom Conservancy? A: Sort of. It’s obviously inspired by them and by a previous announcement by the kernel team which is also inspired by SFC’s principles but doesn’t mention them. The Open Source developer community in general doesn’t just wait 30 days for an infringer to cure their infringement without any penalties. Historically, they’ve waited much longer and the community principles do not specify any particular duration. The infringements of some of my compliance customers would have been impossible to resolve in just 30 days. Some cases have been settled quietly after more than a year in progress, without any cost, or perhaps only a reasonable compliance auditing fee. One of my customers, a Fortune 100 company, was asked to pay $5000 in auditing fees in resolution of a GPL infringement in a Billion dollar product line. Current infringers may be asked for more simply because these cases are expensive to pursue, but in general the community prioritizes compliance over income and does not ask for huge damages as a company in the proprietary software industry would.

Q: Why was today’s announcement made? A: The companies are concerned by one person, Patrick McHardy. I have no personal knowledge of this, but attorney Heather Meeker has documented it. It is said that he has brought about 50 copyright infringement claims regarding the Linux kernel, with intent to collect income rather than simply obtain compliance with the GPL license. Pretty much everyone in the Open Source community and the companies involved all object to this behavior. Me too. But as far as I can tell, it’s McHardy’s legal right to bring such claims regarding the copyrights which he owns, even if it doesn’t fit Community Principles which nobody is actually compelled to follow. The big companies and community members (and I) wish to discourage McHardy and other parties who might wish to become “copyright trolls” in a similar style and exact damages out of unintentional infringers of GPL-liccensed and other software.

Q: Did the Linux Kernel Team make a similar announcement to that of the four companies? A: Yes. Here’s what they had to say. It is probable that in the future the kernel team will stop accepting contributions from people who don’t sign on to the policies at that link. They have presently excluded Mr. McHardy from the kernel team. But it’s important to note that there is lots of existing work in the kernel by copyright holders who are not required to comply with those principles, and the copyright holders all of the GPL software other than the kernel (a larger body of software than the kernel by far) are not required to comply with those principles.

Q: Is it true that the principles the four companies announced today are taken from the GPL 3 license, but they are applying them to GPL 2? A: Yes. If your software is under GPL 3, the same waiting periods that the four companies have promised are required. Thus, it is ironic that when originally presented with the opportunity to apply the GPL 3 to Linux, Linus Torvalds and the Kernel team were quite hostile about it, while the kernel team’s recent announcement attributes the principles they have adopted to the text in GPL 3. Perhaps they’ve learned something since those hostile moments.

Q: I have more questions. A: Send them to bruce at perens dot com.

What Happened to SpaceX’s Zuma Mission? Was It Ever For Real?

The Zuma mission was always mysterious. Now, there’s reason to wonder if it was ever even real.

SpaceX announced on October 16 its mission to launch a secret government satellite built by Northrop-Grumman. It was scheduled to launch within 30 days of the announcement. Keeping a mission secret until the last month before launch was unprecedented for SpaceX, even for previous secret government satellite launches, which had been listed on its launch manifest for years.

Just before Zuma was ready to launch, SpaceX announced that the launch was cancelled via twitter: Standing down on Zuma mission to take a closer look at data from recent fairing testing for another customer.

Subsequently, the Zuma mission was removed from the famous NASA Pad 39a, and returned to the SpaceX garage. SpaceX resumed construction work on Pad 39a to remove the  huge rotating Space Shuttle enclosure and to ready the pad for the three-booster Falcon 9 Heavy demonstration launch. This indicated that there would be no Zuma launch soon. No new launch date has been announced.

So, if we are to take SpaceX’s tweet at face value, SpaceX has suffered a serious failure during testing of its fairing, the “nose cone” of the rocket used to shield a satellite from the atmosphere during launch, which has delayed the Zuma mission until the failure can be understood and corrected.

The next expected SpaceX launch, a cargo launch to ISS expected to fly from Pad 39a on December 4, won’t use the fairing, which is only for satellite launches. Cargo launches to ISS use the Dragon space vehicle. So, this mission would not be delayed.

Since there’s an Iridium satellite launch using the fairing scheduled for December 22 at Vandenberg, we’ll see in a month or so if the fairing problem is so large that it delays further launches. Changes in the fairing could be visible in the launch video, expect space aficionados to pour over that video frame-by-frame.

But there’s another possibility: That the Zuma mission was never real.

Why would SpaceX bring a rocket to the pad on a short schedule only to never launch it? Perhaps the actual mission was to see if SpaceX could put together a launch on a short-enough schedule to satisfy strategic requirements of the government. In tense times a satellite might have to be launched on a short schedule in order to view a country we want a really good look at with special instruments: say, North Korea. Or a failed military satellite might have to be replaced with a standby unit in a big hurry.

If the Zuma mission actually was a test of SpaceX’s ability to launch on a short schedule, they probably passed the test. They put a rocket with what might have been a mock-up satellite on the pad on a short public schedule and test-fired it. We have no idea, of course, of how long SpaceX actually knew about this mission.

And of course the government could have other reasons to delay a spy satellite launch, which we’d never hear of.

SpaceX still has the functioning rocket to satisfy another mission, so if Zuma was never meant to happen, the government might not have had to pay as much as they would for an actual launch. Government launches and launch simulations are still significantly more expensive than civilian ones, due to the vastly increased administrative requirements when working for the government: tests, paperwork, accounting, etc. that a civilian launch customer would not require.

SpaceX fans obsessively track the serial numbers of boosters (seen near the tail fin on the first stage) from the SpaceX factory in California to its test site in Texas to launch in Florida or California. They might catch on if the Zuma booster is eventually used for another mission. Or SpaceX could repaint the serial number to confuse them.

We should know more soon. Perhaps Zuma will launch.