The Soul of an Internet Machine

Engaging a team of software developers requires expertise, patience, and communication between the development team and team that owns and understands needs of the business. There are times that business can operate well with commercial software applications. Some businesses buy multiple tools. At some moment, the leaders within a company acknowledge that their internal or external business workflows are inefficient, or inaccurate, or incompatible with their other technology.  Is there something about your business that sets you apart from the other businesses? What is your competitive advantage? What are the messages you are sending to your clients? Invoicing represents a classic example of this topic. What is an invoice? First definition: It is a document that communicates what a client owes a vendor.During this episode we will explore how an Oracle database can produce a document and we will explore some of the ways that organizations use and design invoices. 

Construction contractors and many other trades love to see scope changes and project expansion. Project expansion and scope changes increase revenue and often increase the size of a team. Sometimes the need for more done at a faster rate comes introduces risk to a project. Using modern software development techniques whereby the team work in 2-4 week sprints, we often see issues long before we must deal with them. How does one solve problems that remain at the horizon? I argue that a mature team works towards the unknown together acknowledging the squishy-bits. We should stand together, client and developer, embracing the challenges – studying them. 

When we start a new software project we “swing with a big hammer”, standing up structures and framework quickly. What we build resembled the both the native Oracle APEX environment plus the client’s colors, logos, and fonts. Through this initial framework, we, the development team, offered a vision. From here a shared vision developed through shared work. We also needed to accommodate the multiple languages of French, Dutch, and the English that the developers depended on. We need to comply with European data privacy laws which are stricter than those in the United States. Building a robust application that accommodates the nuanced complexities that our client will require to differentiate and support their business practices, we design speed and resilience into our system. Designing for speed, also means avoiding techniques that rob us of speed.Speed within a database environment involves honor a series of rules called “Data Normalization”. A discussion about data normalization typically involves slides discussing abstract rules about optimizing complex data within a database environment. Edgar “Ted” Codd, born in 1923, invented the relational model for database management. He worked for IBM. He received the Turing Award in 1981. Dr. Cobb developed the relational model for databases in 1969. Sixty years later, his initial work expanded. The initial data normalization steps I learned have expanded and some nomenclature changed.The amazing thing to me is that people being invented relational databases. And people invented programming languages. Today, we argue about these topics forgetting the humanity underpinning these technologies. My favorite college professor, John Jungck, stood before his Bio 101 course each year to give a lecture called: “There are No Facts”. Unlike modern disputes about facts, he forced no political agenda. He challenged each in the audience to approach science and technology with an open mind. We must eschew assumptions. “Oh, you think two plus two equals four?” Then he reaches under the lectern. He pours two liters of clear water-like fluid into a container with two liters of clear water-like fluid. The total was a bit less than the expected four liters. A bit of chemistry happened given that one of the fluids was not water. Bluntly put, he performed a parlor trick for us. As an illustration, it works. He encouraged student to be curious, skeptical, and carry a bit of doubt when people get dogmatic about anything.All of this technology we use to build and support software applications results from inventions and ideas that came the mind of people. People like you and like me. We invented it all. When honoring the process of normalizing data, we gain speed and maximize performance within relational databases. I do promise, we are talking about how to manage multiple spoken languages within a database. In Episode 2 “Data Tables”, I mentioned that my colleague Dirk provided us with 132 data table definitions filling over 5000 lines of text. I discussed the importance of a unique primary key for each row, or record, of data. One row of data contained the data profile for precisely one subject. One example was the customer table. The customer table has a primary key called the customer_pk. All of the data within that row must related to that exact customer. If it doesn’t related to or describe an element of the client’s profile, then it does not belong. That statement of “fact”, which I surrounded with quotes, complies with the First Normal Form of data. Thank you Doctor Cobb. You documented this idea and stamped it with a name in 1970.When a data row has a singular primary key, as our customer table does with the customer_pk, then it meets the standards for the Second Normal Form. Well done us. This seems obvious to software developers. It seems so obvious to some that we forget the intelligence and humanity behind this concept. Database tables relate to each other. We do not store each and every invoice within the customer table. Instead, we create a table that stores the invoice data. In that table, we’ll have a unique primary key called invoice_pk (meeting both the first and second normal forms – well done us). In the second position of that invoice table, we keep the foreign key for the customer table. That foreign key, our team calls customer_fk. I look at that knowing instantly that invoices relate to customers and the customer foreign key connects to the customer primary key. We break the data into logical and non-duplicating elements. For example, some may opt to put a customer’s address in the customer table. Here’s the street name, the municipality name, the state or province, and the postal code. Suddenly, we find a customer with two addresses. Or maybe one address is for mailing and the other is for shipping. Maybe one address is the physical address, etc. We have all experienced this complexity. When I order from an online vendor, I have one postal code for our physical address and a separate postal code for our mailing address. When I want items shipped to our home, I use the postal code for a municipality that is 50 kilometers east of us. The credit card statement goes to a different postal code which carries a different municipality name. Neither of these agree with the emergency services (or 911/999) address for our farm. That is a third address. Yes, all of these three addresses land on our 40 hectare property clinging to the side of the world’s oldest mountain range. To manage complexity of addresses, we ought to have a table for addresses. The first column is reserved for the address primary key (yes, of course it is called address_pk). The second column is then the customer foreign key (customer_fk). These relationships and efforts to segregate data into a parent/child relationship satisfies the third normal form. Customer is a parent to both address and invoices. In human-speak, one customer may have zero, one, or more addresses. One customer may have zero, one, or more invoices. The parent table may have zero, one, or more related child rows of data in other tables. I have always had difficulty in telling a story about the 4th normal form of relational data. It is a subtle shift in perception of data duplication. In the prior forms, we strove to eliminate the duplication of data within a row. In short, the normalization process discourages us from having fields such as Address 1 and Address 2 and Address 3 and Address 4 in our tables. We should pull that mess out to create an address table, then simplify the customer table. We want to remove these duplicate-like fields of Address 1, Address 2, Address 3. It makes our lives easier. Picture a customer with only 1 address. Can you picture how to handle a customer with four addresses? How do we handle that?In the 4th Normal Form, we strive to reduce redundancy between rows of data. In the Second normal form, we don’t want to keep adding fields to accommodate new and more address for a customer. That’s inefficient and difficult to write. It brings more problems than it solves, Dr Cobb was right. In the 4th Normal Form, we reduce the number of rows by creating related tables. Imagine that each contact at a customer also included the customers address, the customer’s primary phone number, and such. In this example, five rows of contact data would have the same Customer Name. Five rows would have the same address. Five rows would have the same phone numbers. This can result in table having compound keys. The most important data fields are duplicated. When searching for the contacts for Acme Company, I get five rows with the name Acme Company. I see five rows with the same phone number and five rows with the same address. Imagine that two of my contact people have the surname Gonzales. What if a father-son duo work there. We suddenly have Pablo Go...

The first time I heard the phrase middleware, I was writing software for FedEx with a terrific team of Oracle developers in the late 1990s. I had to ask for a definition. In short, middleware is software that run between two things. Middleware is software that is invisible to the user. Middleware is software but seems to fall outside the accepted definitions of an application or app. It is software’s own software-based infrastructure. Does that make any sense? Probably not.I wrote my first lines of code on a PDP-11 microcomputer from Digital Electronics Corporation (DEC) during high school. I attended a school in a wealthy area surrounded by the massively burgeoning IT industry.  In my high school, four years before the IBM-PC came to the market, I learned BASIC. The name literally meant “Beginner’s All-purpose Symbolic Instruction Code”, and no does not relate to the American slang basic, which seems to be a bit of an insult. 10 BEGIN 20 PRINT “HELLO WORLD”30 ENDHard to imagine how these building blocks gave us the world of modern digital commerce. Those roots go back even farther. That trip requires tipping our hat in honor of Ada Lovelace, Alen Turing.My educational progression followed the global development of software and software tools. In 1982, heading to university, I bought an IBM-PC for $2,500. That was the discount a university professor got. Adjusted for 2022 dollars, the cost would approximate $7,500. As a college first year, I thought I would study computer science. Regrettably, my university’s systems were older than my high school’s systems. No way was I going to learn how to program in assembly language on a huge mainframe computer. Mainframes, in 1982, were already doomed, so I thought. I never wanted to write an operating system, which was one of the capstone assignments. I finally took CS-111 as an independent study. In a class by myself, I read Cooper and Clancy’s “Oh Pascal” learning the software language Pascal. That book remains on the shelf behind me in my office today. At the age of 28, I had already been a contributing author and technical editor for books about computer programming. Two evenings a week, I stood in front of a group of adults teaching programming skills at a community college. One high school elective and single independent study course for one semester started me on my career. While an autodidact, nearly all I learned fell beyond the reach of classrooms. School never came easily to me, likely because of my learning differences. I watch colleagues like Dimitri in awe. In direct comparison, I see that I fraud and the idiot, even today. We’re all like that though, aren’t we? We get good at something, or we get recognition then we tell ourselves: No, I don’t deserve this. From my perspective, he is better skilled that I, smarter than I. I admire he jumps between programming languages and environments.I mentioned Pascal on purpose, not just as the rambling digression (I do love my digressions though). The programming language I use for work today had been built from Pascal. Oracle, when needing to create a procedural programming language borrowed heavily from Pascal. Pascal is an imperative and procedural programming language, a natural progression for me from BASIC. The language had been designed by Niklaus Wirth. Mr. Wirth, who is 88 years old in 2022, won the Turing Prize in 1984, roughly the same year I learned Pascal. Stevie and I write code in Oracle’s PL/SQL language. The language derived from an earlier structured procedural language called Pascal, that I learned at university after learning BASIC in high school. We declare variables at the top of a subroutine. We write code in logical subroutines called procedures or functions. These get compiled by Oracle into the database as something that is no longer intelligible to human readers. These routines / subroutines form building blocks within the database to perform tasks, typically with data. One might need a routine that calculates the total value of an invoice. That routine must spin through each line of data for an invoice. The routine identifies the quantity of items ordered. It then multiplies quantity by the unit price to generate the value of that invoice line. Then we tally the total of the invoice lines to get the invoice total. We, the developers, must control this process precisely, due to the variations needed. When do you round the numbers to two decimal places? When do you calculate the taxes? How do you handle items that are not taxed. How to you calculate discounts? Each of these steps must follow the client’s instructions and we must do it precisely. The process and math must be consistent and transparent. Someone will run a calculator through the numbers to confirm the math. Later, an auditor will verify every bit of it. Nothing can be hidden. That’s the land of managing the boring business data: customer contacts, invoice information, inspection data, reporting, document management, bank balances, reservations, etc. My world.Our software depends on a user running the application within a browser. The user updates a customer’s profile or generates a service order. Those tasks such as presenting the data entry web pages to the user’s browser is accomplished by Oracle APEX. I do not have to write the JavaScript and HTML and CSS required to make an application operate within a browser. 98% web-based applications rely on JavaScript. 100% of web-based applications depend on HTML and CSS. We tell the browser to paint a region blue by setting the region’s color property to blue. We do not tell the browser how to do the painting. We only specify the color. We don’t tell a browser how to draw a line, we specify the thickness and style of the line. I won’t present a sample of object-oriented code to read. It doesn’t read well, maybe it is hard to hear: onclick= “void(0);”When I look at our applications written in Oracle APEX, the apps can look different on the different browsers. Occasionally, in Firefox, I see fonts with serif instead of the sans-serif font specified by our team. These variations are reminders of a key difference between procedural languages and object-oriented languages. With object-oriented languages, such as JavaScript, the browser decides how to execute a process. The browser’s own personality expresses itself a bit. One might argue that the process of formatting and presenting data to users via a browser could be called “edge-ware”. Browsers and JavaScript is software sitting at the edge of an application. Edge-ware, a term absolutely nobody uses, sits at the edge of a software application formatting and presenting our pages to the user. Ok, if you do an internet search on Edge-ware, you will find a village in New Zealand and a couple of companies. Let’s also quietly acknowledge that JavaScript is used on servers too. That is a more recent development. The user logs into our Oracle APEX application via browser. The data travels back and forth between the user and the Oracle server via the internet. For illustrative purposes, the Oracle Server sits in the middle of the internet, the “Cloud”. Users sit at the edge of the cloud with their phones, tablets, laptops, and desktop computers. And let’s add cars, refrigerators, doorbell cameras, modern televisions – these all sit at the edge of the cloud with you the user.  The spirit of “edge-ware” tells a story. If we have lovely object-oriented software running on the client-side, or edge, of our applications. And we have lovely software written in a procedural language compiled within the Oracle database, then how does the client/edge talk to the server and the database? Say “Hello” to middleware.Middleware is like a Babelfish (for fans of Hitchhiker’s Guide books)...

I have started so many projects in my life and career. More often, I remember the end of a project. The end of a project means friends disappear. Comfortable familiarity and expertise fades. Sometimes with massive and exhaustive projects, I get sick for a while. When it goes well, I feel snuggly connected to those around me. Leaving Iraq after a year, left me drained, and I failed to keep in tough with others from those days. They also did not keep in touch with me either. That happens after projects end. I did write a colleague from my days at FedEx where we wrote software in Oracle together. This was in the mid to later 1990s. His email had not changed. Our lives had changed during the decades. When revising these teams and times, I find pleasure in the grueling and difficult environment. I find pleasure in remembering the quiet moments and the surprising moments with teammates. In the 2023 series of “The Soul of an Internet Machine”, I am making my second attempt at narrating the beginning of a project. As series 1 informs us, projects fail. Teams fail. Things fall apart. The globe faces a pandemic, etc. In December of 2021, I tempered my enthusiasm about starting a new project with reminders of past failuresOften during years of looking for projects, I see organizations especially our U.S. government looking for software developers. They hire Fred from here and Bugsy from there. Here is Mickey from another place. The bosses say: We need software, so I’ll hire developers. That process often fails. That process certainly costs more money. That process results in immediate, and often systemic problems.Let us explore this together. Electrotest demonstrated immediately the complexity of their demands plus the scope of the project, possibly lasting years. Electrotest, any client or employer, requires us to build them tools that improve their operations. We are expected to improve revenue, reduce expense, improve consistency, reduce regulatory risk, and make the work environment easier on their staff. When the work environment is easier and logical and rhythmic and symmetrical, then training new folks is easier; error rates decrease; and I’ll argue that job satisfaction improves. All too often a bank or a government entity says: Hey, we need to improve our software. The natural result then is hiring programmers. Programmers write software. We want software, therefore we hire programmers. If this is a big project, hire five or ten. If a small project, hire three. The bosses say: we hear Oracle is good. We need Oracle programmers. Or they decide on Microsoft or another brand. The first thing the individual programmers do is introduce themselves. The second thing they do is argue. They argue about standardization, about techniques, about which what is better. We invented a genre of television shows predicated on this experience. The brand “Survivor” comes to mind. One might approach the challenge in one of two ways. I recommend finding a team that had built their credentials and products working together. They arrive with as the “Pros from Dover” often ready to go. They know how to work together. They know each other’s strengths and weaknesses. They possess team shortcuts; team tools; team standards. Their team’s leadership process had been established long before your project. Furthermore, the team tends to success, or fail, together. Their loyalty often focuses on the team instead of their own individual ambitions. We build together, we succeed together. If one of us faces problems, we turn to each other within the team to provide support, love, time, or training – what ever is required. The common choice with many organizations leaps to the conclusion that programmers write software. If you need software, hire programmers. Those assumptions often fail to create an amazing team. Without an amazing cohesive team, you don’t get good software (or a good anything). Good code requires good thinking. Good thinking results in good code. You must hire a team that can think well together, communicate well together, and meet your already impossible deadlines and expectations. Any minute that requires resolving disputes or cleaning up messes costs time, energy, money, and often drains emotions unnecessarily.“Did you do your PSTs?” “Are you walking the dog or is that dog still walking you around?” “Did you find the long pole in the tent?” “I need a rubber-duck.”“Paint with a little brush”“Maintain parallel construction”“Trust the tools”“Baby Steps”“Crawl, walk, run”“Time for a big hammer”“Begin with the end in mind”“Semper Gumby”“Retreat, regroup, return”“Establish a baseline, change one variable, test, repeat.”“A then B then C”“Sometimes good enough is good enough”These phrases form a team’s shorthand. I don’t know the shorthand that Steven Spielberg’s team has. Certainly, a team’s phrases embody the spirit, ethos, and soul of a team. That’s what builds great software. By January 7th or 10th of 2021, we had an operational application running. Technically, we had two applications running. Yet on January 4th, I was still making improvements to the original table structures. I’d make suggestions then beg for approval. That often resulted in difficult phone calls and tension. Dirk and I had never met. We created a demonstration of the worst that happens when you stick two strangers together in a development environment. He was the boss in his mind. I was the boss in my mind. My table designs were better than his. His data table designs came from months of work and numerous meetings with the client for approvals. Dirk and I conflicted continuously. APEX stands at the top of software development tools that are considered low-code/no-code. At the simplest, one can select, drag, and drop data field onto web pages. APEX includes user authentication and user authorization processes. The name APEX derives from a concatenation of Application and Express. APEX resides as a native element within Oracle’s database. It turns a classic server-side database application developer like me to a cool, hip, web-based application developer. I am not required to know CSS nor JavaScript. People create complete applications without writing any code in PL/SQL (Oracles procedure programming language). Nor are folks required to write queries with SQL. This is the definition of low-code/no-code. That takes an application only so far. When you have scores of tables and data for those tables that each need a quick means of management, drag-and-drop is lovely. It is fast. APEX provides standardized and familiar menu systems that resemble the types of menus one sees at on-line shopping sites, banks, and credit cards. I do not have to build that stuff. I don’t care too either. In the early minutes of a project, creating a framework for an application or a suite of interconnected applications ought to be simple, fast, reliable, and consistent. Additionally, I do not want to spent time worrying about the responsiveness of an application. It ought to look great and function fine on a mobile phone platform, a tablet, a laptop, and large-sized monitors often found on an office desk. Somebody else solved that problem already. I don’t need to replicate that. I certainly don’t want to introduce my own mistakes. As the technology related to presenting an application on a web browser improves, I don’t need to chase those improvements around. HTML version 4, then HTML version 5. CSS progressing through the years, now using variable substitutions. Cool, mature, growing up. Yay. Well done HTML and CSS. Oracle’s history spans back to the early 1970s as a relational database tool. There are few c...