Composites with fibers or layers of shape memory alloys (SMA) and elastic or viscoelastic matrix are promising materials for creating multiple-action actuators and power exciters, “artificial muscles”, deployable systems, surfaces of variable geometry, etc. Due to the elastic properties of the matrics, such composites can have the property of repeatedly reversible shape memory, i.e. they can have a two-way shape memory effect, which is not typical for either the binder or the SMA filler without a special thermomechanical processing. With proper design of such a composite, it is possible to achieve the closed property of the two-way shape memory effect, that is, its shape can be controlled in a certain range of changes only by changing the temperature of the filler. Modeling the thermomechanical behavior of composites with elements from SMA is complicated by the complexity of the constitutive relations for SMA, which are of a differential nature, should be considered in the coupled formulation, taking into account the variability of elastic modules of SMA. These constitutive relationships should take into account both the phase and structural mechanism of inelastic deformation of SMA, the fundamental difference between these mechanisms and their mutual influence. This paper uses a variant of the SMA nonlinear deformation model that takes into account all these effects. Polymer binders of composites have not only elastic, but also viscoelastic properties. This circumstance is not taken into account in most works devoted to describing the behavior of composites with elements from SMA. In this paper, the simplest model of linear hereditary viscoelasticity (Kelvin body) is used to evaluate the effect of viscoelastic properties of the matrix on the behavior of a unidirectional composite with SMA fibers. A significant influence of viscoelastic properties on the behavior of the composite at low rates of temperature change of SMA fibers has been established.