Integration guides · 2026-10-17
Jev 1.14 Box: finding an object's location in an image
Find where an object is in an image with Jev 1.14's Box question: typesafe/jev-1.14.0, assets, the experimental header, 0-1 edges and conversion to pixels.
Short answer: a Box question returns the object's four edges
LLMTR (llmtr.com) is a gateway that gives access to language models from different providers through one OpenAI-compatible API. To find where an object is in an image, you call TypeSafe's Jev 1.14 there with the model id typesafe/jev-1.14.0 and a Box question on the POST /v1/systemone endpoint; the base address is https://llmtr.com/v1. The answer gives the four edges of the box around the object as fractions between 0 and 1 of the image's width and height.
Box is one of TypeSafe's experimental features: the provider does not yet call images and Box questions ready for production, and says the request format, answers and limits can change between releases. Box exists only on Jev 1.14 and needs the TypeSafe-Danger-Zone-Images: true header on every request; LLMTR forwards that header to the provider. The model generates no text; 1M input tokens cost $0.042 and output costs $0.
How do you find where an object is in an image through an API?
There are four steps. Encode the image as base64 and add it to the top-level assets map under a reference key; format is png, jpg or jpeg, and data_base64 carries no data: prefix. Put the reference inside the state, for example as the value of a photo key in the state field; the provider places the image wherever the reference appears. In the Box question's instructions, name the thing to find and the image to search; TypeSafe's examples refer to the image by its name in backticks. Finally, send the request with the TypeSafe-Danger-Zone-Images: true header.
The Python example below looks for a coffee mug in a photo of a shelf. The same request also carries a noul question that asks whether a mug is in the image at all, because Box always returns a box. Since the image is in the state, both questions see it. When several questions look at the same image, list it once in the state; using a reference in more than one place adds the image again each time and counts it again. The code converts the box to pixels, and treats the result as not found when there is no confidence value or the presence probability is low.
An example with Python requests that sends a noul and a Box question to typesafe/jev-1.14.0 and converts the box to pixels.
import base64
import os
import requests
from PIL import Image
API_KEY = os.environ["LLMTR_API_KEY"]
PHOTO = "shelf.jpg" # JPEG or PNG, each side 280-1568 px
with open(PHOTO, "rb") as f:
data_base64 = base64.b64encode(f.read()).decode("ascii")
width, height = Image.open(PHOTO).size
response = requests.post(
"https://llmtr.com/v1/systemone",
headers={
"Authorization": f"Bearer {API_KEY}",
"TypeSafe-Danger-Zone-Images": "true",
},
json={
"model": "typesafe/jev-1.14.0",
"state": {"photo": "photo-1"},
"assets": {"photo-1": {"type": "image", "format": "jpg", "data_base64": data_base64}},
"questions": {
"has_mug": {"type": "noul", "instructions": "Is there a coffee mug in `photo`?"},
"mug": {"type": "box", "instructions": "the coffee mug in `photo`"},
},
},
timeout=60,
)
response.raise_for_status()
answers = response.json()["answers"]
mug = answers["mug"]
# 0.5 is an example threshold: tune it on your own images.
if answers["has_mug"]["noul"] < 0.5 or mug["confidence"] is None:
print("No reliable box: treat the mug as not found")
else:
box = mug["box"]
left, right = round(box["left"] * width), round(box["right"] * width)
top, bottom = round(box["top"] * height), round(box["bottom"] * height)
print(f"mug at x {left}-{right}, y {top}-{bottom}, confidence {mug['confidence']}")Reading the answer: edges, pixels and confidence
The box field of a Box answer carries the top, bottom, left and right edges. According to TypeSafe, the origin is the top-left corner: (0, 0) is the top left and (1, 1) the bottom right; left and right are fractions of the width, top and bottom fractions of the height, left is smaller than right and top is smaller than bottom. To get pixels, multiply an edge by the image's width or height: in a 1200 x 800 pixel image, a left of 0.55 puts the box's left edge at pixel 660.
The example response below shows the shape of the request above; the numbers are examples. confidence is between 0 and 1, and according to TypeSafe's API schema it is null when the model gives no confidence. Check for null before you compare with your threshold, and treat a box with a null confidence as undecided. TypeSafe does not yet publish a recommended confidence threshold, so set one on your own images. output_tokens in usage can be above zero, but the price of output is $0.
Example response: the fields follow TypeSafe's documented shape, the numbers are examples.
{
"model": "typesafe/jev-1.14.0",
"answers": {
"has_mug": { "type": "noul", "noul": 0.93 },
"mug": {
"type": "box",
"box": { "top": 0.41, "bottom": 0.62, "left": 0.55, "right": 0.68 },
"confidence": 0.72
}
},
"usage": { "input_tokens": 1180, "output_tokens": 41 }
}There is no not-found answer: practical warnings
Box has no not-found answer. Even when the thing you ask for is not in the image, four edges come back; if you use that box as a real location, your code acts on an object that does not exist. The approach TypeSafe suggests is to ask, in the same request, a noul question about whether the object is in the image, and to ignore the box when the noul answer is low. The 0.5 threshold in the example code is an example.
Each Box question finds one target, and every answer is a single box; to find several things, ask one Box question for each. If the image holds several things of the same kind, describe the one you want, such as the mug on the left or the mug with the red lid. If words cannot tell them apart, TypeSafe suggests adding a choice question with none, one and several, and when the answer is several, splitting the image into regions. For a low-confidence box, it suggests asking again with a larger version of the image or a higher-resolution crop around the box, or checking the edges with noul questions. TypeSafe warns that a target a few dozen pixels wide in a very large image can be missed; for small targets, crop the image or split it into tiles before sending it.
Image rules, limits and error codes
The image must be PNG or JPEG, between 280 and 1568 pixels on each side, and at most 16 MiB; according to TypeSafe, an image outside these limits is rejected, never resized to fit. LLMTR accepts images only in a JSON body, as data_base64; multipart uploads are not accepted, and the request body is limited to 24 MB in total. Base64 encoding makes a file about a third larger. Every image you list must be referred to at least once.
A file over 16 MiB or not a PNG or JPEG, a Box question with no image it can see, and a Box question sent to the typesafe/jev id (Jev 1.13) return 400 invalid_request before the request reaches the model. A missing header or image dimensions outside the limits return 400 provider_error, and the message says what is wrong. Image tokens count as input at the same price as text and arrive in usage.input_tokens. TypeSafe has not yet published a guide for estimating image tokens, so track the cost from the usage value of your own requests.
| Rule | Value |
|---|---|
| Model id | typesafe/jev-1.14.0 |
| Required header | TypeSafe-Danger-Zone-Images: true |
| Image format | PNG or JPEG, as data_base64 |
| Image side | 280-1568 pixels |
| Image file | at most 16 MiB |
| Request body | at most 24 MB |
| Input price (images included) | $0.042 per 1M tokens |
| Output price | $0 per 1M tokens |
Find where an object is in an image with Jev 1.14
Add the image to the assets map, ask a noul and a Box question, and convert the box to pixels.
- Preparation. Create a key under Dashboard > API Keys, top up your balance under Dashboard > Billing, and pick a PNG or JPEG whose sides are 280-1568 pixels.
- Request. Add the image to the assets map as data_base64, put its reference in the state, define a noul and a Box question, and send the request for typesafe/jev-1.14.0 to https://llmtr.com/v1/systemone with the TypeSafe-Danger-Zone-Images: true header.
- Reading. Ignore the box when the noul answer is low or confidence is null; otherwise convert the edges to pixels by multiplying them by the image's width and height.
Frequently asked questions
Which model answers Box questions?
On LLMTR only typesafe/jev-1.14.0 (Jev 1.14) answers Box questions, and the request needs the TypeSafe-Danger-Zone-Images: true header. A Box question sent to another model returns 400 invalid_request before the request reaches the model.
What comes back if the object is not in the image?
A box still comes back; Box has no not-found answer. Ask a noul question in the same request about whether the object is in the image, and ignore the box when that probability is low.
How do I convert Box coordinates to pixels?
Multiply left and right by the image's width, and top and bottom by its height. The values are fractions between 0 and 1, counted from the top-left corner.