The Schillace Laws - Part 1: Taming the Beast of Large Language Models

A year ago, if someone had told me that a simple text file with a natural language prompt would become an integral part of my software system, I would have raised an eyebrow. It's even more surprising to think that formulating a question or a request to an AI model would be a crucial step in my development process. But here we are :) So, fasten your seatbelts, and let's dive into The Schillace Laws!

The early days

As developers, we're constantly discovering and embracing tools, practices, and concepts that enhance our professional lives. IDEs like Visual Studio, version control systems like Git, testing tools such as NUnit, frameworks and libraries tailored to specific use cases like ASP.NET and Laravel, and design patterns and principles like SOLID and GoF Design Patterns—all aimed at increasing productivity, improving collaboration, and ensuring code quality and reliability.

But have you ever considered the parallels between the early days of programming languages and the emergence of "prompt engineering," specifically the fusion of conventional programming languages with prompts?

Before the advent of these robust toolsets, we relied on writing code in text editors and compilers to translate it into binary code for execution. While we've come a long way from punch cards and paper tapes, it's fascinating to draw connections between those early programming days and the current landscape of prompt engineering.

Let's explore how these well-established tools align with the advancements in LLMs, as well as the unique challenges they present.

The Power of IDEs, Version Control, Testing, Frameworks and Patterns

Regarding IDEs, we recently witnessed an exciting development at the Microsoft Build conference: the introduction of Prompt Flow, a feature of AzureML. This innovative feature is a promising step in the right direction for integrating prompts seamlessly into our development workflows.

Fig.1 - Prompt creation with AzureML Prompt Flow.

When it comes to version control, we're fortunate that traditional VC systems like Git work effectively with prompts. After all, a prompt is merely text, making it easy to manage versioning and collaboration.

However, in the realm of testing, we still face challenges. Presently, it's difficult to switch between models confidently and ensure that our samples produce the expected results. While Prompt Flow shows promise as a visual testing tool, we eagerly anticipate advancements in testing libraries and frameworks that will efficiently tackle this challenge.

Frameworks are a bit different since the true power of prompts and LLMs lies in combining them with conventional languages. Thankfully, we already have SDKs like Semantic Kernel, Langchain, and others that enable this fusion, opening up a world of possibilities.

Design Patterns and Principles: Bridging the Gap

Conventional programming comes with a plethora of design patterns, principles, best practices, and reference architectures. These serve as valuable guidebooks for deterministic code.

However, with prompts, we face a level of vagueness. To address this, Sam Schilacce, deputy CTO of Microsoft, working extensively with various Large Language Model (LLM) AI systems, formulated a set of principles known as The Schillace Laws. These principles offer invaluable guidance to developers who are accustomed to established best practices and principles, helping us tame the beast called LLM.

John Maeda, in his article "Early Lessons From GPT-4: The Schillace Laws" divided these laws into three distinct groups:

Relationships Between Code And Models

1. Don't write code if the model can do it; the model will get better, but the code won't.

2. Code is for syntax and process; models are for semantics and intent.

3. Text is the universal wire protocol.

Trade-offs And Challenges When Using LLMs

4. Trade leverage for precision; use interaction to mitigate.

5. The system will be as brittle as its most brittle part.

6. Uncertainty is an exception throw.

7. Hard for you is hard for the model.

Limitations and Opportunities of Using LLMs

8. Ask Smart to Get Smart.

9. Beware "pareidolia of consciousness"; the model can be used against itself.

In this blog series, we will delve into each group of Schillace Laws, exploring practical examples that demonstrate how to implement these principles in practice. By understanding and leveraging these laws, we can navigate the realm of LLMs more effectively and harness their potential to transform our development workflows.

Check out the next post in this series, where we'll explore the group Relationships Between Code And Models and illustrate real-world scenarios that showcase the application of these laws. Together, let's tame the beast of Large Language Models!

Stay tuned and happy coding!

The Schillace Laws - Part 1: Taming the Beast of Large Language Models

The Schillace Laws - Part 2: Relationships Between Code And Models

The Schillace Laws - Part 3: Trade-offs And Challenges When Using LLMs

The Schillace Laws - Part 4: Limitations and Opportunities of Using LLMs