What distinguishes software architecture from coding and design? Many things, including the role that architects have in defining architectural characteristics, the important aspects of the system independent of the problem domain. Many organizations describe these features of software with a variety of terms, including nonfunctional requirements, but we dislike that term because it is self-denigrating. Architects created that term to distinguish architecture characteristics from functional requirements, but naming something nonfunctional has a negative impact from a language standpoint: how can teams be convinced to pay enough attention to something “nonfunctional”? Another popular term is quality attributes, which we dislike because it implies after-the-fact quality assessment rather than design. We prefer architecture characteristics because it describes concerns critical to the success of the architecture, and therefore the system as a whole, without discounting its importance.

An architecture characteristic meets three criteria:

• Specifies a nondomain design consideration
• Influences some structural aspect of the design
• Is critical or important to application success

Source: Mark Richards and Neal Ford. Software Architecture 2024 V2 . Kindle Edition.

For work I have a mandatory secure development course that includes a web security lab. PortSwigger has a broad array of details topics on web security covering the ones you'd expect but also advanced, newer topics like Web LLM attacks.

One thing I did learn was that, even though I have multi-factor authentication set up for AWS, it's not truly multi-factor. I use a password and a passkey but they both are provided by password manager using my fingerprint. It's very convenient though. Maybe I should continue getting a second factor from my phone's authenticator app, which is slightly more time consuming, but more secure.

Here are a few way I explore CSV files, with DuckDB being a new entrant.

CSVKit

I find myself working with CSV files these days. I sometimes need to look around inside. For 10s of rows, a text editor will do - I just use VS Code. csvlook from csvkit is great too.

The following command will allow me to scroll through the first 500 lines of a file. The --no-inference flag informs csvlook to not infer data type and format values accordingly.

head -n 500 yourfile.csv | csvlook --no-inference | less -S

Sometimes I find myself needing to filter out rows. The following command will filter rows based on the presence of \\N in any of the columns between 1 and 12, and output the inverse (-i) results (good data) to a file.

csvgrep -c 1-12 -m "\\N" -i -a file_with_unwanted_data.csv > file-with_unwanted_rows_removed.csv

SQLite

Sometimes I want to explore the file a bit and csvkit tools becoming a bit unwieldy. SQLlite is perfect for the job. Remember, SQLite is included in macOS and Mac OS X by default, so sqlite-utils will just work once installed.

This command switches to CSV mode and imports the CSV file into a table called prices:

sqlite>.mode csv
sqlite>.import prices.csv prices

Julia Evans mentioned sqlite-utils in her blog post sqlite-utils: a nice way to import data into SQLite for analysis It's another tool from the prolific Simon Willison.

The following command installs sqlite-utils and uses it to create a new databases called prices.db (sqlite is a file-based database) with a table called prices with records inserted into the table from the CSV file:

brew install sqlite-utils

Once installed, you can import with this command:

sqlite-utils insert prices.db prices prices.csv --csv

With large files, Simon notes sqlite-utils can be slower compared to using SQLite directly. You can import JSON and other formats just as easily.

Once you have a SQLite database, use standard SQL to query.

DuckDB

I just listed to a podcast on Software Engineering Daily about DuckDB. The founder, Hannes Mühleisen, makes a claim that the DuckDB team have inadvertently built the worlds best CSV reader.

duckdb -c  "select * from read_csv('prices.csv')"

DuckDB doesn't need to ingest the data into its database engine. There are clients for various programming languages. And you can read from many different data formats beyond CSV. Paul Gross has started to use DuckDB as a replacement for jq.

Complexity is the gotcha of event-driven architecture. David Boyne talks about the inevitable complexity of EDAs at part of GOTO's most recent EDA Day.

What was new to me was the The project paradox which is summed up by the question "Why do we take the biggest decisions at a time when we have the least knowledge?".

EDAs ease this because they are evolutionary by nature but can get complex quickly if not managed, which you do by documenting your events and architecture.

programming for kids. Borislav has been teaching his 10 year old daughter how to code. He views programming as something to be literate in, not something to have a career in necessarily. I agree. I wonder how E will respond if I try to teach him a bit every day. Borislav has kept a record of each day's lesson. He has managed to teach on 403 days so far! Wow, what a gift to his daughter.

I noticed the senior engineers at work didn't have a space to come together. Senior engineers are distributed over many client projects and, from experience, they too need a support group. I, as a tech principal, need their help elevating the the technical capabilities across the organisation.

Here's the process I took:

• validate what my senses were telling me
  • On an individual basis, I told each senior engineer what I was thinking and asked whether they felt a need for a space for senior engineers to connect.
  • Each engineer responded positively with their own personal take
• invited senior engineers to discuss further
  • shared the positive responses anonymously in an email invitation
• prepared a one pager to guide our discussion, including steps for:
  • context setting
  • purpose - potential topics, initiatives, opportunities,
  • membership - who was in this group and who was not
  • structure - how often, the formats (round table, lean coffee, knowledge sharing, etc)
  • needs - what might the needs be of the group
  • roles & responsibilities - rotation of various roles, e.g. facilitator, note-taker, knowledge curators, etc...
• have the discussion for 45 minutes

The pace of the session went well, we covered all the steps, I made sure all voices were heard, and it felt like everyone was on the same page.

We agreed to meet fortnightly, have a mixture of formats, limit membership to level 5 engineers and above to keep group small and manageable. There were a lot of ideas covered when we spoke about purpose of the group and needs of the members, including:

• cohesive set of standards for common tech we use
• port of call for feedback on strategic objectives
• working out what messages to amplify to wider engineering discipline
• peer support
• fostering sense of belonging and collaboration outside of client work
• coordination of talent development, training needs and cultural change

All in all, a very worthwhile session that I should have made happen sooner.

At work, I've enrolled for a programme called Inclusion Mentoring. After an hour of an explainer, I was paired with a colleague who was different enough from me so we could both learn new perspectives. Different backgrounds, different teams and different identities.

Session one was about finding out about each others background and agreeing a charter that makes sense to both of us. We agreed that there's no hierarchy, we will swap from mentee to mentor numerous times in a conversation and we everything that's said will remain confidential. I spoke about coming from a broken home, my teenage years, summarised my 20s and 30s, and where I'm at right now in my mid 40s. I also highlighted my privileges: race: white; gender: cis male; physical ability: able-bodied; class: wealthy (relatively but grew up working class); nationality: western; orientation: hetero. Plus two other bits of my identity: religion: nothing organised, maybe spiritual; mental ability: neurodiverse (self diagnosis).

We got into some excellent topics and I learnt a bunch of new perspectives. I shan't go into them here otherwise I'll betray confidence if someone reads from work.

If you get the chance to do this, go for it. You'll learn a lot from a deep, focused conversation.

Over the past few weeks I've been using a 5k speed improvement program suggested by my Corus watch. It's:

• Warm up: 3.2km at threshold pace of 69% to 81%, which is my aerobic endurance pace
• 8 sets of
  • 200m at threshold pace of > 114%, which gets me into anaerobic power
  • 3 minutes recovery pace, basically walking
• 1.6km threshold pace of between 92% and 102%
• Cool down: 2.4km threshold pace of 69% to 81%

I've been doing this a few weeks now. Yesterday, I switched to place in my local park which had a flat strip that was near enough 200m, so I could visually see the finishing line approaching, making me keep the pace as fast as I could. On the 4th time, a muscle fibre in my left hamstrung went twang. I tried gently running after a few minutes but it felt uncomfortable. Very annoying.

After reading a few blogs, I think I have a grade 1 strain where there is "There is little structural disruption, and the muscle heals quickly".

There are a bunch of exercises I should do on https://www.sportsinjuryclinic.net/sport-injuries/thigh-pain/back-thigh/pulled-hamstring-exercises. When I can run again, I should not sprint as fast as I can. I should gradually increase the speed over many weeks, whilst at the same time strengthening my hamstrings so they can handle the extra load sprints put on to the muscle.

I wanted to use Simon Willison's shot-scraper tool. This made me understand how to set up a Python dev environment.

First, I upgrade Python:

~ % which python3
/opt/homebrew/bin/python3
~ % python3 --version
Python 3.12.2
~ % brew upgrade python3
~ % python3 --version
Python 3.12.3

Then I attempt to install shot-scraper:

screenshots % pip install shot-scraper
zsh: command not found: pip

I try pip3 install shot-scraper. This works, but doesn't install the package:

screenshots % pip3 install shot-scraper
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz
    
    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with
    
    brew install pipx
    
    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.
    
    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.
    
    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Claude gives me some options:

The error message you're getting is because of the way Homebrew manages and isolates Python packages to prevent conflicts with the system's Python installation.

Here are a few options to address this issue:

1. Use Homebrew to install Python packages

You can use Homebrew to install Python packages by running:

brew install <package_name>

Replace <package_name> with the package you want to install. This will install the package in a separate Homebrew-managed location, avoiding conflicts with the system's Python.

2. Create and use a virtual environment

Virtual environments are isolated Python environments that allow you to install packages without affecting your system's Python installation or other virtual environments. To create and use a virtual environment, follow these steps:

python3 -m venv /path/to/your/venv
source /path/to/your/venv/bin/activate
pip install <package_name>

Replace /path/to/your/venv with the desired location for your virtual environment, and <package_name> with the package you want to install. After activating the virtual environment, you can use pip to install packages within that isolated environment.

3. Use pipx (recommended)

pipx is a tool that automatically creates virtual environments and installs packages in an isolated location. It's a convenient way to manage Python packages without interfering with your system's Python installation. Install pipx first:

brew install pipx
pipx ensurepath

Then, you can install packages with pipx:

pipx install <package_name>

This will create a virtual environment for the package and add the package's executable to your PATH, making it easy to run.

By following one of these approaches, you can install Python packages on your macOS system without running into conflicts with Homebrew's managed Python installation.

I've going to try pipx, although I'm not entirely sure what it is:

pipx install shot-scraper

Whilst it's installing, I look for more info about pipx. Its README says:

pipx is a tool to help you install and run end-user applications written in Python. It's roughly similar to macOS's brew, JavaScript's npx, and Linux's apt.

I know what npx is so I now have a good mental model. Jacob adds to my understanding:

Why? pipx lets me install Python-based CLI stuff (youtube-dl, awscli, doc2dash, etc.) without those projects’ dependencies messing up my global Python.

shot-scraper has installed successfully:

  installed package shot-scraper 1.4, installed using Python 3.12.3
  These apps are now globally available
    - shot-scraper
done! ✨ 🌟 ✨

I use YouTube Premium, which is ad-free, plus other features.

Problem

On my work Macbook, when I click on a YouTube.com link it opens in Chrome, my default browser, which is logged into my work Google Workspace account. Work obviously doesn't pay for YouTube Premium so I have to watch ads or copy the link to a different browser where I'm logged into YouTube Premium via my personal Google Account.

I don't want to copy links between browsers.

Solution

https://github.com/johnste/finicky