Use new URL parameter to send attachments

[simonw]

https://twitter.com/alexalbert__/status/1895504248206709246

We've added the ability to specify a public facing URL as the source for an image / document block in the Anthropic API

[simonw]

I am going to manually test this to make sure it works for both images and PDFs.

[simonw]

I expect this will speed up conversation mode a whole lot, since it won't need to include base64 images in every turn of the conversation.

[simonw]

Based on these docs looks like PDF URLs are supported: https://docs.anthropic.com/en/api/messages#body-messages-content

[simonw]

Relevant commit to Anthropic Python SDK: anthropics/anthropic-sdk-python@9048173

[simonw]

I need at least https://github.com/anthropics/anthropic-sdk-python/releases/tag/v0.48.0

[simonw]

I added a pprint(messages) to help test this:

llm -m claude-3.5-haiku describe -a https://static.simonwillison.net/static/2024/pelican.jpg
[{'content': [{'source': {'type': 'url',
                          'url': 'https://static.simonwillison.net/static/2024/pelican.jpg'},
               'type': 'image'},
              {'text': 'describe', 'type': 'text'}],
  'role': 'user'}]
This image shows a brown pelican perched on some rocks near a boat. The pelican has a distinctive long beak and grayish-brown plumage. Its feathers look slightly ruffled, and it appears to be sitting in a maritime setting, with a white boat visible in the background. The lighting suggests it's during daytime, with a soft, slightly blurred background that keeps the focus on the pelican. The bird seems calm and stationary, typical of pelicans when they're resting or observing their surroundings. The rocky foreground and the boat create a classic coastal or marina scene.

llm -c 'what are some details you missed?'
[{'content': [{'source': {'type': 'url',
                          'url': 'https://static.simonwillison.net/static/2024/pelican.jpg'},
               'type': 'image'},
              {'text': 'describe', 'type': 'text'}],
  'role': 'user'},
 {'content': 'This image shows a brown pelican perched on some rocks near a '
             'boat. The pelican has a distinctive long beak and grayish-brown '
             'plumage. Its feathers look slightly ruffled, and it appears to '
             'be sitting in a maritime setting, with a white boat visible in '
             "the background. The lighting suggests it's during daytime, with "
             'a soft, slightly blurred background that keeps the focus on the '
             'pelican. The bird seems calm and stationary, typical of pelicans '
             "when they're resting or observing their surroundings. The rocky "
             'foreground and the boat create a classic coastal or marina '
             'scene.',
  'role': 'assistant'},
 {'content': 'what are some details you missed?', 'role': 'user'}]
Great question! Here are some details I missed in my first description:

1. Color nuance: The pelican's feathers aren't just grayish-brown, but have a more complex coloration with varying shades of gray, white, and brownish-gray.

2. Precise body posture: The pelican is actually perched with a very specific, upright stance, with its long neck curved slightly.

3. Head details: The pelican has a white/off-white head coloration that contrasts with its body, which I didn't highlight before.

4. Beak detail: The beak is not just long, but has a distinctive shape with a slight downward curve and appears to have a reddish or pinkish tint at the tip.

5. Lighting specifics: The image has a soft, slightly diffused light that creates gentle shadows and highlights the texture of the pelican's feathers.

6. Background depth: While I mentioned the boat, I didn't note that the boat is slightly out of focus, creating depth in the image.

7. Rocks: The rocks in the foreground are not just generic rocks, but look like oyster shells or similar marine terrain.

These additional details provide a more nuanced and complete description of the image. Thank you for prompting me to look closer!

[simonw]

Trying something very challenging:

llm -m claude-3.7-sonnet -a 'https://tile.loc.gov/storage-services/service/ndnp/mb/batch_mb_gaia_ver02/data/sn86086481/0051717161A/1900012901/0296.pdf' \
  'transcribe all text from this image, formatted as markdown'

https://tile.loc.gov/storage-services/service/ndnp/mb/batch_mb_gaia_ver02/data/sn86086481/0051717161A/1900012901/0296.pdf is a big complex image:

It output this from the pprint:

[{'content': [{'source': {'type': 'url',
                          'url': 'https://tile.loc.gov/storage-services/service/ndnp/mb/batch_mb_gaia_ver02/data/sn86086481/0051717161A/1900012901/0296.pdf'},
               'type': 'document'},
              {'text': 'transcribe all text from this image, formatted as '
                       'markdown',
               'type': 'text'}],
  'role': 'user'}]

And then went ahead and did a really great job. Truncated:

MUCH SUFFERING

Capt. Abercrombie Tells of the Horrors Along the Copper River

SEVENTY PER CENT OF PEOPLE INSANE

Scurvy Killing Hundreds—Commander of Copper River Expedition Reports on Routes For Government Roads

Washington, Jan. 28.—The war department has made public an important report from Capt. W. R. Abercrombie, Second Infantry, who commanded the Copper River exploring expedition in Alaska last season. The chief topic treated in the report is the laying out of the great trans-Alaskan military route from Port Valdez, Alaska, to Fort Egbert on the Yukon. While engaged in this work, the officer was charged to take note of the mineral resources of the country, of its adaptability for agriculture and stock raising, and the fuel and food products for man and animal.

[...]

Full response: https://gist.github.com/simonw/df1a0473e122830d55a0a3abb51384c9

Total cost 16.3242 cents.

[simonw]

And just for curiosity, following https://llm.datasette.io/en/stable/schemas.html#extracting-people-from-a-news-articles

llm -m claude-3.7-sonnet -a 'https://tile.loc.gov/storage-services/service/ndnp/mb/batch_mb_gaia_ver02/data/sn86086481/0051717161A/1900012901/0296.pdf' \
  -t people
```json
{
  "items": [
    {
      "name": "Captain W. R. Abercrombie",
      "organization": "US Army, Second Infantry",
      "role": "Commander of the Copper River exploring expedition in Alaska",
      "learned": "Reported on severe suffering and health conditions along the Copper River, with 70% of people mentally deranged and many suffering from scurvy and frost-bite. Also reported on routes for government roads in Alaska.",
      "article_headline": "MUCH SUFFERING: Capt. Abercrombie Tells of the Horrors Along the Copper River",
      "article_date": "1900-01-29"
    },
    {
      "name": "Edward Gillette",
      "organization": "Copper River exploring expedition",
      "role": "Member of the expedition",
      "learned": "Contributed information on the feasibility of establishing a practical railroad route up the Copper River valley, comparing the route from Seattle to Skaguay with the Valdez route.",
      "article_headline": "MUCH SUFFERING: Capt. Abercrombie Tells of the Horrors Along the Copper River",
      "article_date": "1900-01-29"
    }
  ]
}

I think a bunch of people are missing from that.

Replaying the OCR output text directly into that template:

llm logs --cid 01jn7h45x2dafa34zk30z7ayfy -r | llm -t people -m claude-3.7-sonnet

Gave this:

{
  "items": [
    {
      "name": "Capt. W. R. Abercrombie",
      "organization": "United States Army",
      "role": "Commander of Copper River exploring expedition",
      "learned": "Reported on the horrors along the Copper River in Alaska, including starvation, scurvy, and mental illness affecting 70% of people. He was tasked with laying out a trans-Alaskan military route and assessing resources.",
      "article_headline": "MUCH SUFFERING",
      "article_date": "1900-01-28"
    },
    {
      "name": "Edward Gillette",
      "organization": "Copper River expedition",
      "role": "Member of the expedition",
      "learned": "Contributed a chapter to Abercrombie's report on the feasibility of establishing a railroad route up the Copper River valley, comparing it favorably to the Seattle to Skaguay route.",
      "article_headline": "MUCH SUFFERING",
      "article_date": "1900-01-28"
    },
    {
      "name": "Gen. Buller",
      "organization": "British Army",
      "role": "General",
      "learned": "Had been down with fever but recovered, according to a Boer report. Was leading British forces in the conflict against Boers.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Gen. Woodgate",
      "organization": "British Army",
      "role": "General leading infantry",
      "learned": "Led British infantry in storming Tabanyama, a key point of Boer position, on January 24.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Gen. Warren",
      "organization": "British Army",
      "role": "General",
      "learned": "His troops captured a portion of Tabanyama mountain but faced difficulty holding it against Boer forces.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Dr. Leyds",
      "organization": "Transvaal Government",
      "role": "Diplomat/Envoy",
      "learned": "Was being warmly received in Berlin and attempting to persuade Germany to mediate in the conflict, offering commercial and mining monopolies. Said the war would last a very long time and Transvaal would not seek peace first.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Dr. Emil Pretorius",
      "organization": "The Post",
      "role": "Editor",
      "learned": "Acted as chairman at a pro-Boer meeting in St. Louis where resolutions were passed calling for U.S. intervention.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "W. J. Stone",
      "organization": "Former Missouri government",
      "role": "Ex-Governor",
      "learned": "Gave an address at a pro-Boer meeting in St. Louis.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Gov. John Lind",
      "organization": "Minnesota government",
      "role": "Governor",
      "learned": "Delivered an address at a pro-Boer mass meeting in Minneapolis attended by 4,000 people.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Mayor James Gray",
      "organization": "Minneapolis government",
      "role": "Mayor",
      "learned": "Delivered an address at a pro-Boer mass meeting in Minneapolis.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Judge William Prentiss",
      "organization": "Chicago judiciary",
      "role": "Judge",
      "learned": "Delivered an address at a pro-Boer mass meeting in Minneapolis.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Sir Alfred Milner",
      "organization": "British Government",
      "role": "British high commissioner",
      "learned": "Issued a proclamation that the British government would not recognize property forfeitures in Transvaal or Free State after October 10, when war was declared.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "James Jubb",
      "organization": "Unknown",
      "role": "British citizen in America",
      "learned": "An Englishman who committed suicide in New Britain, Connecticut, allegedly due to insanity caused by constantly brooding over the war in Transvaal.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Lord Roberts",
      "organization": "British Army",
      "role": "Commander",
      "learned": "Visited the hospital ship Maine in Cape Town and expressed satisfaction with the arrangements.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Lady Randolph Churchill",
      "organization": "Hospital ship Maine",
      "role": "Supervisor/Organizer",
      "learned": "Conducted Lord Roberts on a tour of the hospital ship Maine in Cape Town.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "John Churchill",
      "organization": "South African Light Horse",
      "role": "Officer",
      "learned": "Second son of Lady Randolph Churchill, who received his commission in the South African Light Horse from Lord Roberts.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Dr. Jameson",
      "organization": "British forces",
      "role": "Military doctor",
      "learned": "Reportedly wounded in the leg as a result of Boer artillery fire on Monday.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Pres. Steyn",
      "organization": "Orange Free State",
      "role": "President",
      "learned": "Visited the Boer laager (camp) near Ladysmith.",
      "article_headline": "LADYSMITH TO BE LEFT TO HER FATE",
      "article_date": "1900-01-29"
    },
    {
      "name": "Gen. Mercier",
      "organization": "French military/government",
      "role": "General and newly elected Senator",
      "learned": "Was elected to the French Senate, defeating M. Rano. He was a Dreyfus persecutor and a prominent Nationalist candidate.",
      "article_headline": "MERCIER ELECTED",
      "article_date": "1900-01-29"
    },
    {
      "name": "Gen. Lambert",
      "organization": "French military/government",
      "role": "General and newly elected Senator",
      "learned": "One of only two military men running as Nationalist candidates who succeeded in being elected to the French Senate.",
      "article_headline": "MERCIER ELECTED",
      "article_date": "1900-01-29"
    },
    {
      "name": "M. Rano",
      "organization": "Dreyfusard movement",
      "role": "Political candidate",
      "learned": "A Dreyfusard who was defeated by Gen. Mercier in the French Senate election. His defeat was greeted with counter-cries in the streets.",
      "article_headline": "MERCIER ELECTED",
      "article_date": "1900-01-29"
    },
    {
      "name": "M. Bourgeois",
      "organization": "French government",
      "role": "Former Prime Minister",
      "learned": "Referenced by the newspaper Figaro as having had his ministry forced to resign by the Senate.",
      "article_headline": "MERCIER ELECTED",
      "article_date": "1900-01-29"
    },
    {
      "name": "George White",
      "organization": "Unknown",
      "role": "Farmer",
      "learned": "A well-known farmer in Uxbridge who fell down a flight of stairs in his barn and was instantly killed.",
      "article_headline": "A FATAL FALL",
      "article_date": "1900-01-28"
    },
    {
      "name": "E. J. Phelps",
      "organization": "U.S. Government (former)",
      "role": "Ex-minister to England",
      "learned": "Was in critical condition though physicians were reported to be hopeful about his recovery.",
      "article_headline": "Phelps' Case Hopeful",
      "article_date": "1900-01-28"
    },
    {
      "name": "J. H. Althans",
      "organization": "Bakery business",
      "role": "Bakery owner",
      "learned": "His bakery in Providence at the corner of Brook and John Streets was practically destroyed by fire, with losses of about $20,000.",
      "article_headline": "Blaze in Providence",
      "article_date": "1900-01-28"
    },
    {
      "name": "James W. Alexander",
      "organization": "Princeton Club",
      "role": "Princeton alumnus, class of 1860",
      "learned": "Recited a parody poem entitled \"Poe's Tiger\" at a \"smoker\" event honoring Princeton's football team.",
      "article_headline": "\"POE'S TIGER\"",
      "article_date": "1900-01-29"
    }
  ]
}