We propose an attitude filtering and magnetometer calibration approach for nanosatellites. Measurements from magnetometers, Sun sensor and gyros are used in the filtering algorithm to estimate the attitude of the satellite together with the bias terms for the gyros and magnetometers. In the traditional approach for the attitude filtering, the attitude sensor measurements are used in the filter with a nonlinear vector measurement model. In the proposed algorithm, the TRIAD algorithm is used in conjunction with the unscented Kalman filter (UKF) to form the nontraditional attitude filter. First the vector measurements from the magnetometer and Sun sensor are processed with the TRIAD algorithm to obtain a coarse attitude estimate for the spacecraft. In the second phase the estimated coarse attitude is used as quaternion measurements for the UKF. The UKF estimates the fine attitude, and the gyro and magnetometer biases. We evaluate the algorithm for a hypothetical nanosatellite by numerical simulations. The results show that the attitude of the satellite can be estimated with an accuracy better than 0.5 and the computational load decreases more than 25% compared to a traditional UKF algorithm. We discuss the algorithm's performance in case of a time-variance in the magnetometer errors.